[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:57 dist-test-slave-8bgz krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:57 dist-test-slave-8bgz krb5kdc[399](info): set up 2 sockets
Jun 19 20:54:57 dist-test-slave-8bgz krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 20:54:58.158466 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:58.268476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:58.347404 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:58.347584 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:58.347606 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:58.348739 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:58.348780 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:58.348795 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:58.348846 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:58.348867 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:58.350360 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:58.350600 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:58.350833 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:58.355912 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.355947 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.355957 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:58.355955 412 server_base.cc:1034] running on GCE node
I20260619 20:54:58.356349 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:58.356640 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:58.357813 412 hybrid_clock.cc:648] HybridClock initialized: now 1781902498357802 us; error 25 us; skew 500 ppm
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:58.360173 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:58.360602 412 webserver.cc:460] Webserver started at http://127.0.98.62:33081/ using document root <none> and password file <none>
I20260619 20:54:58.360778 412 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:58.360817 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:58.360901 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:58.361901 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "5d12d98f8b12497896280b689ec76785"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "eea05297dda838e45508920ba1566315"
server_key_iv: "e4653ef07439e61e6dcb62ca9688df6d"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.362291 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d12d98f8b12497896280b689ec76785"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "eea05297dda838e45508920ba1566315"
server_key_iv: "e4653ef07439e61e6dcb62ca9688df6d"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.363531 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:58.364216 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.364406 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:58.364493 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "5d12d98f8b12497896280b689ec76785"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "eea05297dda838e45508920ba1566315"
server_key_iv: "e4653ef07439e61e6dcb62ca9688df6d"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.364563 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:58.376034 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:58.377708 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:58.377876 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:58.381601 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41395
I20260619 20:54:58.381628 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41395 every 8 connection(s)
I20260619 20:54:58.381976 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:58.382745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 412
I20260619 20:54:58.382864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:58.383050 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:58.383747 392 external_mini_cluster.cc:1428] Setting key c48a78bdf78212ce7f22b8218b7c493f
I20260619 20:54:58.386531 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785: Bootstrap starting.
I20260619 20:54:58.387255 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:58.387573 480 log.cc:824] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:58.388345 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785: No bootstrap required, opened a new log
I20260619 20:54:58.390831 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260619 20:54:58.390980 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:58.391024 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d12d98f8b12497896280b689ec76785, State: Initialized, Role: FOLLOWER
I20260619 20:54:58.391144 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260619 20:54:58.391223 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:58.391263 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:58.391309 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:58.391842 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260619 20:54:58.391956 480 leader_election.cc:302] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d12d98f8b12497896280b689ec76785; no voters:
I20260619 20:54:58.392139 480 leader_election.cc:288] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:58.392165 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:58.392304 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [term 1 LEADER]: Becoming Leader. State: Replica: 5d12d98f8b12497896280b689ec76785, State: Running, Role: LEADER
I20260619 20:54:58.392395 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:58.392490 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260619 20:54:58.392858 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d12d98f8b12497896280b689ec76785. Latest consensus state: current_term: 1 leader_uuid: "5d12d98f8b12497896280b689ec76785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260619 20:54:58.393014 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:58.393285 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d12d98f8b12497896280b689ec76785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d12d98f8b12497896280b689ec76785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260619 20:54:58.393359 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:58.393919 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:58.394547 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:58.398625 494 catalog_manager.cc:1347] Generated new cluster ID: 77c79319500d40d2b6fe02448d4e515e
I20260619 20:54:58.398687 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:58.400532 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.385147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.62:41395)
0619 20:54:58.386334 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.386342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.386377 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:54:58.387481 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.387487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.387530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.387678 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.389676 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.390711 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.391894 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.392521 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.392613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.396902 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.396905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.396906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.396913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.398692 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.399156 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.399159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.399159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.399231 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.399561 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.399562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.399562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.399791 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.399922 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.400197 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.400290 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":69,"threads_started":1}
I20260619 20:54:58.416196 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:58.416875 494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:58.426493 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d12d98f8b12497896280b689ec76785: Generated new TSK 0
I20260619 20:54:58.426678 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:54:58.454813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:58.567839 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:58.568105 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:58.568145 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:58.569741 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:58.569784 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:58.569820 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:54:58.571774 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:58.571979 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:58.572191 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:54:58.577375 506 server_base.cc:1034] running on GCE node
W20260619 20:54:58.577322 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.577314 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.577332 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:58.577709 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:58.577939 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:58.579105 506 hybrid_clock.cc:648] HybridClock initialized: now 1781902498579095 us; error 28 us; skew 500 ppm
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:58.581409 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:54:58.581861 506 webserver.cc:460] Webserver started at http://127.0.98.1:34523/ using document root <none> and password file <none>
I20260619 20:54:58.582063 506 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:58.582124 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:58.582283 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:58.583323 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "012fc713a9f245e09cefb508a2c43bff"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "f26f46827e18a0b5598a6992b3a63d6c"
server_key_iv: "c2859e2e693edbe0101454c1d3dd1617"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.583686 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "012fc713a9f245e09cefb508a2c43bff"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "f26f46827e18a0b5598a6992b3a63d6c"
server_key_iv: "c2859e2e693edbe0101454c1d3dd1617"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.584937 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:54:58.585608 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.585774 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:58.585880 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "012fc713a9f245e09cefb508a2c43bff"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "f26f46827e18a0b5598a6992b3a63d6c"
server_key_iv: "c2859e2e693edbe0101454c1d3dd1617"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.585964 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:58.601290 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:58.603320 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:58.603466 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:58.603693 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:54:58.603972 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:54:58.604023 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.604063 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:54:58.604087 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.609668 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45949
I20260619 20:54:58.609745 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45949 every 8 connection(s)
I20260619 20:54:58.610037 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:58.619174 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.610817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59197 (local address 127.0.98.62:41395)
0619 20:54:58.610861 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.610863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.611076 (+ 213us) server_negotiation.cc:407] Connection header received
0619 20:54:58.611577 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.611579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.611590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.611641 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.612582 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.613081 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.614106 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.614415 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.614493 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.616566 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.616567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.616568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.616576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.617832 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.618253 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.618255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.618255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.618313 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.618554 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.618555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.618555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.618666 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.618741 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.619001 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.619084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:54:58.619346 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.610786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:59197)
0619 20:54:58.610957 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:58.610989 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:58.611472 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:58.611717 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:58.611721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:58.612303 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:58.612477 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.612482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.613176 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.613177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.613999 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.614000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.614498 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.614499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.614631 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.615109 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:58.615116 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:58.616467 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:58.617928 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.617935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.617937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.618149 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.618382 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.618383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.618384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.618486 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.618739 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:58.618745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:58.618875 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 20:54:58.618980 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:54:58.619128 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260619 20:54:58.619663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 506
I20260619 20:54:58.619761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:54:58.619941 392 external_mini_cluster.cc:1428] Setting key d8456ca854328a9f73a043b8998c1746
I20260619 20:54:58.619951 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260619 20:54:58.620035 635 heartbeater.cc:464] Registering TS with master...
I20260619 20:54:58.620219 635 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260619 20:54:58.620760 444 ts_manager.cc:194] Registered new tserver with Master: 012fc713a9f245e09cefb508a2c43bff (127.0.98.1:45949)
I20260619 20:54:58.621337 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59197
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:54:58.643150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:58.732895 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:58.733114 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:58.733150 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:58.734335 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:58.734392 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:58.734437 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:54:58.735873 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:58.736159 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:58.736388 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:58.741210 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.741218 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:58.741278 640 server_base.cc:1034] running on GCE node
W20260619 20:54:58.741218 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:58.741621 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:58.741852 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:58.742995 640 hybrid_clock.cc:648] HybridClock initialized: now 1781902498742976 us; error 34 us; skew 500 ppm
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:58.745218 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:54:58.745680 640 webserver.cc:460] Webserver started at http://127.0.98.2:37349/ using document root <none> and password file <none>
I20260619 20:54:58.745898 640 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:58.745990 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:58.746121 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:58.747133 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "667e6fb19e9a4c42bf56ce935519e308"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "9f862c1e45f796d9642765c4d13f4948"
server_key_iv: "5a5b96555ff5b3fc242adc06f7759fe2"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.747493 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "667e6fb19e9a4c42bf56ce935519e308"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "9f862c1e45f796d9642765c4d13f4948"
server_key_iv: "5a5b96555ff5b3fc242adc06f7759fe2"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.748745 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:54:58.749399 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.749570 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:54:58.749660 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "667e6fb19e9a4c42bf56ce935519e308"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "9f862c1e45f796d9642765c4d13f4948"
server_key_iv: "5a5b96555ff5b3fc242adc06f7759fe2"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.749742 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:58.761662 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:58.763423 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:58.763581 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:58.763774 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:54:58.764070 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:54:58.764118 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.764150 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:54:58.764174 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.770399 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34829
I20260619 20:54:58.770455 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34829 every 8 connection(s)
I20260619 20:54:58.770761 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:58.777546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 640
I20260619 20:54:58.777621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:54:58.777840 392 external_mini_cluster.cc:1428] Setting key b5ac06346fddbcf34e0d4feefb156362
I20260619 20:54:58.780411 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.771880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60317 (local address 127.0.98.62:41395)
0619 20:54:58.771940 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.771943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.772176 (+ 233us) server_negotiation.cc:407] Connection header received
0619 20:54:58.772598 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.772599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.772608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.772657 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.773488 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.773951 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.774851 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.775025 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.775085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.777261 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.777262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.777262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.777268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.779123 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.779480 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.779481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.779482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.779542 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.779750 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.779751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.779751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.779896 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.779968 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.780289 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.780350 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:54:58.780673 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.771853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:60317)
0619 20:54:58.772063 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:58.772096 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:58.772522 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:58.772705 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:58.772708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:58.773235 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:58.773422 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.773426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.774020 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.774021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.774779 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.774780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.775101 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.775101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.775183 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.775583 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:58.775590 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:58.777185 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:58.779199 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.779204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.779205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.779410 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.779604 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.779605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.779605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.779692 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.779980 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:58.779985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:58.780146 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 20:54:58.780261 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 20:54:58.780433 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260619 20:54:58.781152 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260619 20:54:58.781224 769 heartbeater.cc:464] Registering TS with master...
I20260619 20:54:58.781414 769 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260619 20:54:58.781785 444 ts_manager.cc:194] Registered new tserver with Master: 667e6fb19e9a4c42bf56ce935519e308 (127.0.98.2:34829)
I20260619 20:54:58.782232 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60317
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:54:58.802420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:58.880432 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:58.880613 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:58.880635 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:58.881743 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:58.881785 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:58.881841 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:54:58.883208 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:58.883435 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:58.883649 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:58.888406 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.888451 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:58.888417 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:58.888613 774 server_base.cc:1034] running on GCE node
I20260619 20:54:58.888826 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:58.889032 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:58.890175 774 hybrid_clock.cc:648] HybridClock initialized: now 1781902498890165 us; error 40 us; skew 500 ppm
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:58.893008 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:54:58.893605 774 webserver.cc:460] Webserver started at http://127.0.98.3:44173/ using document root <none> and password file <none>
I20260619 20:54:58.893891 774 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:58.893990 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:58.894125 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:58.895422 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e205580d70094e70865ad318824c2ad3"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "b67bc4d2dc064b08638f425fdd290d00"
server_key_iv: "207382a53fad67692a894a68532a0bb5"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.895864 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e205580d70094e70865ad318824c2ad3"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "b67bc4d2dc064b08638f425fdd290d00"
server_key_iv: "207382a53fad67692a894a68532a0bb5"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.897430 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:58.898100 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.898262 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:58.898355 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e205580d70094e70865ad318824c2ad3"
format_stamp: "Formatted at 2026-06-19 20:54:58 on dist-test-slave-8bgz"
server_key: "b67bc4d2dc064b08638f425fdd290d00"
server_key_iv: "207382a53fad67692a894a68532a0bb5"
server_key_version: "encryptionkey@0"
I20260619 20:54:58.898406 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:58.912227 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:58.913808 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:58.913911 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:58.914093 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:54:58.914428 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:54:58.914462 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.914484 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:54:58.914496 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:58.920984 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35761
I20260619 20:54:58.921056 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35761 every 8 connection(s)
I20260619 20:54:58.921388 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:54:58.926757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 774
I20260619 20:54:58.926851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:54:58.927011 392 external_mini_cluster.cc:1428] Setting key 9c51eef8f62c612249a56875f703272a
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:58.930387 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.922135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52155 (local address 127.0.98.62:41395)
0619 20:54:58.922184 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.922187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.922490 (+ 303us) server_negotiation.cc:407] Connection header received
0619 20:54:58.922936 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.922936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.922946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.922999 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.923881 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.924403 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.925367 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.925552 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.925615 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.927971 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.927973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.927973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.927980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.929239 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.929599 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.929600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.929600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.929649 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.929850 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.929851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.929851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.929945 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.929994 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.930199 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.930328 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:58.930583 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.922127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:52155)
0619 20:54:58.922329 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:58.922403 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:58.922857 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:58.923054 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:58.923056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:58.923618 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:58.923805 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.923810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.924483 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.924484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.925264 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:58.925265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.925628 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.925629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:58.925721 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.926200 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:58.926265 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:58.927867 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:58.929303 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.929308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.929310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.929534 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.929707 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:58.929707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:58.929708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:58.929797 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:58.929998 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:58.930002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:58.930110 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:54:58.930188 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:54:58.930374 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260619 20:54:58.930999 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260619 20:54:58.931085 903 heartbeater.cc:464] Registering TS with master...
I20260619 20:54:58.931213 903 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260619 20:54:58.931560 444 ts_manager.cc:194] Registered new tserver with Master: e205580d70094e70865ad318824c2ad3 (127.0.98.3:35761)
I20260619 20:54:58.931907 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52155
I20260619 20:54:58.939039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:54:58.960903 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.953540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.0.98.1:45949)
0619 20:54:58.953689 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.953694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.953712 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:54:58.953735 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.953736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.953749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.953819 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.954542 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.955215 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.955887 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.956184 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.956236 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.958388 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.958395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.958395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.958402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.959983 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.960299 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.960301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.960301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.960368 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.960515 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.960515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.960516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.960622 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.960720 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.960773 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.960829 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260619 20:54:58.961283 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:54:58 dist-test-slave-8bgz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:54:58.981395 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:58.974921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.98.1:45949)
0619 20:54:58.974982 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:58.974984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:58.975418 (+ 434us) server_negotiation.cc:407] Connection header received
0619 20:54:58.975468 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:58.975469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:58.975478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:58.975526 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:58.976008 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.976459 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.977113 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:58.977311 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:58.977358 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:58.979431 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:58.979432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:58.979432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:58.979437 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:58.980533 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.980836 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.980837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.980837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.980881 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:58.981034 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:58.981035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:58.981035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:58.981111 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:58.981194 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:58.981258 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 20:54:58.981312 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:54:58.981912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 506
I20260619 20:54:58.986407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 640
I20260619 20:54:58.991808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 774
I20260619 20:54:58.995915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 412
2026-06-19T20:54:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1884 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:59 dist-test-slave-8bgz krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:59 dist-test-slave-8bgz krb5kdc[917](info): set up 2 sockets
Jun 19 20:54:59 dist-test-slave-8bgz krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 20:55:00.017764 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:00 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:00.128696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:00.209627 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:00.209882 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:00.209923 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:00.211107 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:00.211181 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:00.211210 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:00.211244 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:00.211275 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:00.212733 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:00.213012 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:00.213260 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:00.218384 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:00.218387 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:00.218552 930 server_base.cc:1034] running on GCE node
W20260619 20:55:00.218675 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:00.218958 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:00.219208 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:00.220377 930 hybrid_clock.cc:648] HybridClock initialized: now 1781902500220369 us; error 30 us; skew 500 ppm
Jun 19 20:55:00 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:00.222944 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:00.223413 930 webserver.cc:460] Webserver started at http://127.0.98.62:41239/ using document root <none> and password file <none>
I20260619 20:55:00.223634 930 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:00.223677 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:00.223822 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:00.224898 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "353e1654284849139a6f7562ef21174b"
format_stamp: "Formatted at 2026-06-19 20:55:00 on dist-test-slave-8bgz"
server_key: "2de1947eabb45bc1ebfc02625f097547"
server_key_iv: "1756bcf3d6923f60fc6719c48e28b86c"
server_key_version: "encryptionkey@0"
I20260619 20:55:00.225302 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "353e1654284849139a6f7562ef21174b"
format_stamp: "Formatted at 2026-06-19 20:55:00 on dist-test-slave-8bgz"
server_key: "2de1947eabb45bc1ebfc02625f097547"
server_key_iv: "1756bcf3d6923f60fc6719c48e28b86c"
server_key_version: "encryptionkey@0"
I20260619 20:55:00.226804 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:00.227595 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:00.227788 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:00.227876 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "353e1654284849139a6f7562ef21174b"
format_stamp: "Formatted at 2026-06-19 20:55:00 on dist-test-slave-8bgz"
server_key: "2de1947eabb45bc1ebfc02625f097547"
server_key_iv: "1756bcf3d6923f60fc6719c48e28b86c"
server_key_version: "encryptionkey@0"
I20260619 20:55:00.227950 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:00.253787 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:00.255901 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:00.256053 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:00.260206 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39179
I20260619 20:55:00.260243 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39179 every 8 connection(s)
I20260619 20:55:00.260576 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:00.261402 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:00.263125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 930
I20260619 20:55:00.263213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:00.263371 392 external_mini_cluster.cc:1428] Setting key 07cbbe54819e71ebc1d6284875235f6d
I20260619 20:55:00.264063 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b: Bootstrap starting.
I20260619 20:55:00.264628 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:00.264982 998 log.cc:824] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:00.265756 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b: No bootstrap required, opened a new log
I20260619 20:55:00.267957 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260619 20:55:00.268172 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:00.268216 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353e1654284849139a6f7562ef21174b, State: Initialized, Role: FOLLOWER
I20260619 20:55:00.268343 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260619 20:55:00.268399 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:00.268419 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:00.268479 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 0 FOLLOWER]: Advancing to term 1
Jun 19 20:55:00 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:00.273384 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:00.263874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.62:39179)
0619 20:55:00.264042 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:00.264049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:00.264073 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:55:00.264522 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:00.264528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:00.264573 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:00.264720 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:00.265444 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:00.266465 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:00.267540 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:00.267936 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:00.268014 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:00.270546 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:00.270549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:00.270550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:00.270559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:00.272177 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:00.272451 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:00.272452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:00.272453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:00.272520 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:00.272654 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:00.272655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:00.272655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:00.272871 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:00.272977 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:00.273067 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:55:00.273194 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260619 20:55:00.352883 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260619 20:55:00.353055 998 leader_election.cc:302] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353e1654284849139a6f7562ef21174b; no voters:
I20260619 20:55:00.353366 998 leader_election.cc:288] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:00.353526 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:00.353760 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [term 1 LEADER]: Becoming Leader. State: Replica: 353e1654284849139a6f7562ef21174b, State: Running, Role: LEADER
I20260619 20:55:00.353778 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:00.353960 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260619 20:55:00.354460 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353e1654284849139a6f7562ef21174b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260619 20:55:00.354602 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:00.354987 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353e1654284849139a6f7562ef21174b. Latest consensus state: current_term: 1 leader_uuid: "353e1654284849139a6f7562ef21174b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e1654284849139a6f7562ef21174b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260619 20:55:00.355109 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:00.355551 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:00.356037 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:00.358291 1012 catalog_manager.cc:1347] Generated new cluster ID: 3eb4a690f1094cfca5554bc1a82b2405
I20260619 20:55:00.358352 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:00.369123 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:00.369853 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:00.383878 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353e1654284849139a6f7562ef21174b: Generated new TSK 0
I20260619 20:55:00.384070 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260619 20:55:01.828635 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.434s user 0.000s sys 0.002s
I20260619 20:55:01.828961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:01.909905 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:01.910252 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:01.910295 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:01.911443 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:01.911479 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:01.911505 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:01.912838 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:01.913050 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:01.913307 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:01.918618 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:01.918618 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:01.918805 1024 server_base.cc:1034] running on GCE node
W20260619 20:55:01.918931 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:01.919121 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:01.919366 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:01.920534 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781902501920520 us; error 29 us; skew 500 ppm
Jun 19 20:55:01 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:01.922933 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:01.923393 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40447/ using document root <none> and password file <none>
I20260619 20:55:01.923616 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:01.923682 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:01.923832 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:01.924860 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33"
format_stamp: "Formatted at 2026-06-19 20:55:01 on dist-test-slave-8bgz"
server_key: "524d1b9bfe4bd081fce501240c98d8d7"
server_key_iv: "1ad52764919523a77001ddcf57f3db91"
server_key_version: "encryptionkey@0"
I20260619 20:55:01.925217 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33"
format_stamp: "Formatted at 2026-06-19 20:55:01 on dist-test-slave-8bgz"
server_key: "524d1b9bfe4bd081fce501240c98d8d7"
server_key_iv: "1ad52764919523a77001ddcf57f3db91"
server_key_version: "encryptionkey@0"
I20260619 20:55:01.926532 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:01.927285 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:01.927474 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:01.927588 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33"
format_stamp: "Formatted at 2026-06-19 20:55:01 on dist-test-slave-8bgz"
server_key: "524d1b9bfe4bd081fce501240c98d8d7"
server_key_iv: "1ad52764919523a77001ddcf57f3db91"
server_key_version: "encryptionkey@0"
I20260619 20:55:01.927660 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:01.942842 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:01.944797 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:01.944967 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:01.945217 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:01.945518 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:01.945576 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:01.945617 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:01.945647 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:01.951591 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40399
I20260619 20:55:01.951666 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40399 every 8 connection(s)
I20260619 20:55:01.951992 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:01.953644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1024
I20260619 20:55:01.953732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:01.953904 392 external_mini_cluster.cc:1428] Setting key 786731b1d461faabd6cf2b0e26b2f2fd
Jun 19 20:55:01 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:01.962706 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:01.952717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53403 (local address 127.0.98.62:39179)
0619 20:55:01.952870 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:01.952872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:01.953023 (+ 151us) server_negotiation.cc:407] Connection header received
0619 20:55:01.953668 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:01.953669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:01.953680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:01.953756 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:01.955098 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:01.955682 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:01.956676 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:01.957021 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:01.957094 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:01.959798 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:01.959799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:01.959799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:01.959805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:01.961243 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:01.961732 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:01.961732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:01.961733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:01.961784 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:01.962054 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:01.962054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:01.962055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:01.962163 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:01.962293 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:01.962574 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 20:55:01.962639 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260619 20:55:01.963011 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:01.952703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:53403)
0619 20:55:01.952883 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:01.952927 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:01.953514 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:01.953848 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:01.953852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:01.954787 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:01.954998 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:01.955005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:01.955771 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:01.955772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:01.956566 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:01.956568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:01.957121 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:01.957122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:01.957262 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:01.957777 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:01.957787 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:01.959707 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:01.961325 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:01.961330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:01.961332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:01.961651 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:01.961848 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:01.961849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:01.961849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:01.961984 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:01.962262 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:01.962269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:01.962428 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 20:55:01.962557 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 20:55:01.962736 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:55:01.963591 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260619 20:55:01.963696 1153 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:01.963863 1153 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260619 20:55:01.964366 962 ts_manager.cc:194] Registered new tserver with Master: ce213cf2d4ab4df1ad0d39b97efd4c33 (127.0.98.1:40399)
I20260619 20:55:01.964967 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 20:55:02.965875 1153 heartbeater.cc:503] Master 127.0.98.62:39179 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 20:55:02.971257 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.017s user 0.001s sys 0.001s
I20260619 20:55:02.971624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:03.052623 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:03.052819 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:03.052841 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:03.054544 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:03.054594 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:03.054625 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:03.056226 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:03.056458 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:03.056682 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:03.062076 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:03.062227 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:03.062110 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:03.062423 1159 server_base.cc:1034] running on GCE node
I20260619 20:55:03.062644 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:03.062875 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:03.064018 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781902503063998 us; error 34 us; skew 500 ppm
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:03.071204 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:03.071684 1159 webserver.cc:460] Webserver started at http://127.0.98.2:36289/ using document root <none> and password file <none>
I20260619 20:55:03.071905 1159 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:03.071971 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:03.072127 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:03.073160 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "dcdfab560308499fb8f607e8fc73b901"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "56224fecc83ebca0b9e5d767d3b7b59f"
server_key_iv: "b84bb6370a4d737ddaff0b3735ac36a5"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.073518 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dcdfab560308499fb8f607e8fc73b901"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "56224fecc83ebca0b9e5d767d3b7b59f"
server_key_iv: "b84bb6370a4d737ddaff0b3735ac36a5"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.074939 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:03.075686 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.075871 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:03.075963 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "dcdfab560308499fb8f607e8fc73b901"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "56224fecc83ebca0b9e5d767d3b7b59f"
server_key_iv: "b84bb6370a4d737ddaff0b3735ac36a5"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.076018 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:03.105530 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:03.107434 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:03.107545 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:03.107720 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:03.108024 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:03.108058 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.108080 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:03.108129 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.113837 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42627
I20260619 20:55:03.113893 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42627 every 8 connection(s)
I20260619 20:55:03.114199 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:03.116784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1159
I20260619 20:55:03.116878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:03.117044 392 external_mini_cluster.cc:1428] Setting key 7c0865c6e214968a93cffd4df99d9fb5
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:03.124650 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.114964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53431 (local address 127.0.98.62:39179)
0619 20:55:03.115067 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.115069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.115274 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:55:03.115776 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.115777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.115788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.115846 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.116919 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.117609 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.119193 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.119520 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.119588 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.121671 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.121673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.121673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.121681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.123232 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.123668 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.123669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.123669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.123731 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.123978 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.123979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.123979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.124103 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.124179 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.124467 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.124551 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
I20260619 20:55:03.124821 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.114926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:53431)
0619 20:55:03.115124 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.115160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.115688 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.115901 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.115904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.116510 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:03.116782 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.116789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.117694 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.117695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.119084 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.119088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.119598 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.119599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.119695 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.120218 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:03.120226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:03.121576 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:03.123333 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:03.123339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:03.123341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:03.123580 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:03.123797 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:03.123797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:03.123798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:03.123909 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:03.124179 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:03.124185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:03.124323 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.124435 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.124590 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:55:03.125262 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260619 20:55:03.125373 1288 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:03.125552 1288 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260619 20:55:03.125929 962 ts_manager.cc:194] Registered new tserver with Master: dcdfab560308499fb8f607e8fc73b901 (127.0.98.2:42627)
I20260619 20:55:03.126395 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:03.141453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:03.219272 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:03.219457 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:03.219486 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:03.220628 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:03.220667 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:03.220726 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:03.222106 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:03.222354 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:03.222569 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:03.227385 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:03.227361 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:03.227479 1294 server_base.cc:1034] running on GCE node
W20260619 20:55:03.227362 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:03.227759 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:03.227945 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:03.229113 1294 hybrid_clock.cc:648] HybridClock initialized: now 1781902503229101 us; error 25 us; skew 500 ppm
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:03.231803 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:03.232231 1294 webserver.cc:460] Webserver started at http://127.0.98.3:35333/ using document root <none> and password file <none>
I20260619 20:55:03.232474 1294 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:03.232517 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:03.232654 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:03.233675 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "46f8b53b6d1446c794ff4d5601baff74"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "7cfba9926a68d172948a81cb1a52eb5f"
server_key_iv: "4eade2adabab00aaaa36cbcb5d0abab7"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.234022 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "46f8b53b6d1446c794ff4d5601baff74"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "7cfba9926a68d172948a81cb1a52eb5f"
server_key_iv: "4eade2adabab00aaaa36cbcb5d0abab7"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.235291 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:03.235996 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.236156 1294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:03.236250 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "46f8b53b6d1446c794ff4d5601baff74"
format_stamp: "Formatted at 2026-06-19 20:55:03 on dist-test-slave-8bgz"
server_key: "7cfba9926a68d172948a81cb1a52eb5f"
server_key_iv: "4eade2adabab00aaaa36cbcb5d0abab7"
server_key_version: "encryptionkey@0"
I20260619 20:55:03.236338 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:03.244714 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:03.246549 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:03.246699 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:03.246908 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:03.247210 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:03.247262 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.247303 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:03.247331 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:03.253145 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43619
I20260619 20:55:03.253213 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43619 every 8 connection(s)
I20260619 20:55:03.253556 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:03.255771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1294
I20260619 20:55:03.255867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:03.256028 392 external_mini_cluster.cc:1428] Setting key 56d183b84042fb58bea0abe13078c175
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:03.263039 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.254322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34781 (local address 127.0.98.62:39179)
0619 20:55:03.254431 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.254433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.254642 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:55:03.255135 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.255136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.255144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.255199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.256320 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.257035 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.257991 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.258189 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.258365 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.260444 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.260446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.260446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.260455 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.261825 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.262203 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.262251 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.262251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.262309 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.262531 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.262532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.262532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.262630 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.262681 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.262904 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.262980 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 20:55:03.263190 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.254329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:34781)
0619 20:55:03.254510 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.254551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.255052 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.255249 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.255252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.255957 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:03.256234 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.256239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.257104 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.257106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.257892 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.257894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.258371 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.258372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.258477 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.258948 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:03.258957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:03.260292 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:03.261909 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:03.261914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:03.261916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:03.262122 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:03.262369 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:03.262370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:03.262370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:03.262473 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:03.262684 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:03.262688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:03.262795 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.262879 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.263006 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260619 20:55:03.263688 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260619 20:55:03.263785 1423 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:03.263973 1423 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260619 20:55:03.264365 962 ts_manager.cc:194] Registered new tserver with Master: 46f8b53b6d1446c794ff4d5601baff74 (127.0.98.3:43619)
I20260619 20:55:03.264678 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34781
I20260619 20:55:03.267616 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:55:03.274789 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.269074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.0.98.62:39179)
0619 20:55:03.269166 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.269168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.269318 (+ 150us) server_negotiation.cc:407] Connection header received
0619 20:55:03.269345 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.269346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.269357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.269416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.270304 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.270806 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.271466 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.271635 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.271683 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.272754 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.272755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.272755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.272760 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.273843 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.274125 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.274126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.274126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.274172 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.274393 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.274393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.274393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.274513 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.274586 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.274678 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.274731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 20:55:03.276681 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:03.278185 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:03.284597 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.98.1:40399)
0619 20:55:03.280813 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.280818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.280944 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:55:03.281090 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.281092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.281108 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.281198 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:03.281324 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282269 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.283637 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284202 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284239 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284276 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.284482 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.284521 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260619 20:55:03.284597 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40399 (local address 127.0.0.1:46066)
0619 20:55:03.280828 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.280869 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.280988 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.281198 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.281201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.281203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:03.281262 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.281264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282353 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.282354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.283542 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.283544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284324 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.284375 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284383 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.284467 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.284518 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260619 20:55:03.284822 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42627 (local address 127.0.0.1:51886)
0619 20:55:03.280945 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.280954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.281047 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.281301 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.281302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.281304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:03.281352 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.281354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282728 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.282729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.283575 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.283578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284430 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.284512 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284517 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.284550 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.284591 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":86,"threads_started":1}
I20260619 20:55:03.284874 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.2:42627)
0619 20:55:03.280918 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.280922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.281034 (+ 112us) server_negotiation.cc:407] Connection header received
0619 20:55:03.281127 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.281129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.281148 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.281220 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:03.281430 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282603 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.283637 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284328 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284665 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284723 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.284748 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.284792 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260619 20:55:03.285430 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 55917b48d20c43acb2218f72c83ce713 (DEFAULT_TABLE table=test-table [id=b24c7eb83674429ebbd9c92bfee72733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:03.285566 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 55917b48d20c43acb2218f72c83ce713 (DEFAULT_TABLE table=test-table [id=b24c7eb83674429ebbd9c92bfee72733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:03.285776 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55917b48d20c43acb2218f72c83ce713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:03.285892 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55917b48d20c43acb2218f72c83ce713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:03.286332 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.0.98.3:43619)
0619 20:55:03.280904 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.280909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.281236 (+ 327us) server_negotiation.cc:407] Connection header received
0619 20:55:03.281274 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.281277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.281292 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.281364 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:03.281493 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282600 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.283825 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284521 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284628 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.284860 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.284905 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260619 20:55:03.286455 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.280495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43619 (local address 127.0.0.1:34678)
0619 20:55:03.281126 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.281135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.281179 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.281361 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.281362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.281364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:03.281410 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.281411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.282704 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.282705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.283748 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.283749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.284659 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.284660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.284722 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.284726 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.284775 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.284805 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260619 20:55:03.287096 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 55917b48d20c43acb2218f72c83ce713 (DEFAULT_TABLE table=test-table [id=b24c7eb83674429ebbd9c92bfee72733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:03.287397 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55917b48d20c43acb2218f72c83ce713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:03.288336 1441 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Bootstrap starting.
I20260619 20:55:03.289165 1442 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: Bootstrap starting.
I20260619 20:55:03.289451 1441 tablet_bootstrap.cc:654] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:03.289814 1442 tablet_bootstrap.cc:654] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:03.290045 1441 log.cc:824] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:03.290112 1442 log.cc:824] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:03.290866 1442 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: No bootstrap required, opened a new log
I20260619 20:55:03.290942 1442 ts_tablet_manager.cc:1397] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:03.291028 1441 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: No bootstrap required, opened a new log
I20260619 20:55:03.291090 1441 ts_tablet_manager.cc:1397] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 20:55:03.292984 1441 raft_consensus.cc:348] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.293170 1441 raft_consensus.cc:374] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:03.293205 1441 raft_consensus.cc:729] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdfab560308499fb8f607e8fc73b901, State: Initialized, Role: FOLLOWER
I20260619 20:55:03.293318 1441 consensus_queue.cc:260] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.293406 1442 raft_consensus.cc:348] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.293558 1442 raft_consensus.cc:374] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:03.293596 1441 ts_tablet_manager.cc:1428] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:55:03.293605 1442 raft_consensus.cc:729] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce213cf2d4ab4df1ad0d39b97efd4c33, State: Initialized, Role: FOLLOWER
I20260619 20:55:03.293622 1288 heartbeater.cc:503] Master 127.0.98.62:39179 was elected leader, sending a full tablet report...
I20260619 20:55:03.293712 1442 consensus_queue.cc:260] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.294075 1442 ts_tablet_manager.cc:1428] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:03.294269 1445 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: Bootstrap starting.
I20260619 20:55:03.294930 1445 tablet_bootstrap.cc:654] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:03.295416 1445 log.cc:824] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:03.296049 1445 tablet_bootstrap.cc:492] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: No bootstrap required, opened a new log
I20260619 20:55:03.296100 1445 ts_tablet_manager.cc:1397] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:03.297772 1445 raft_consensus.cc:348] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.297894 1445 raft_consensus.cc:374] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:03.297950 1445 raft_consensus.cc:729] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f8b53b6d1446c794ff4d5601baff74, State: Initialized, Role: FOLLOWER
I20260619 20:55:03.298075 1445 consensus_queue.cc:260] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.298362 1445 ts_tablet_manager.cc:1428] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:55:03.298393 1423 heartbeater.cc:503] Master 127.0.98.62:39179 was elected leader, sending a full tablet report...
I20260619 20:55:03.345465 1449 raft_consensus.cc:482] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:03.345577 1449 raft_consensus.cc:504] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.345875 1449 leader_election.cc:288] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcdfab560308499fb8f607e8fc73b901 (127.0.98.2:42627), ce213cf2d4ab4df1ad0d39b97efd4c33 (127.0.98.1:40399)
I20260619 20:55:03.348681 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.346034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40399 (local address 127.0.98.3:59585)
0619 20:55:03.346176 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.346190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.346256 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.346419 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.346420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.346421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:03.346488 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.346490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.347147 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.347148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.347908 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.347910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.348425 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.348506 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.348508 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.348574 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.348613 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":26}
I20260619 20:55:03.348680 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.346260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59585 (local address 127.0.98.1:40399)
0619 20:55:03.346308 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.346309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.346318 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:03.346344 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.346345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.346357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.346413 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:03.346569 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.347043 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348006 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.348348 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.348435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.348579 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.348614 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:03.348798 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.346159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35307 (local address 127.0.98.2:42627)
0619 20:55:03.346222 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.346224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.346445 (+ 221us) server_negotiation.cc:407] Connection header received
0619 20:55:03.346465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.346466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.346475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.346533 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:03.346694 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.347169 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348081 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.348462 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348547 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.348556 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.348697 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.348733 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:55:03.348814 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.346034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42627 (local address 127.0.98.3:35307)
0619 20:55:03.346229 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:03.346238 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:03.346294 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:03.346547 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:03.346547 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:03.346549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:03.346625 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.346626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.347249 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.347250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.347995 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:03.347997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.348548 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.348549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:03.348624 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.348627 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:03.348692 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:55:03.348730 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":11,"thread_start_us":101,"threads_started":1}
I20260619 20:55:03.349040 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55917b48d20c43acb2218f72c83ce713" candidate_uuid: "46f8b53b6d1446c794ff4d5601baff74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" is_pre_election: true
I20260619 20:55:03.349217 1107 raft_consensus.cc:2454] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f8b53b6d1446c794ff4d5601baff74 in term 0.
I20260619 20:55:03.349320 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55917b48d20c43acb2218f72c83ce713" candidate_uuid: "46f8b53b6d1446c794ff4d5601baff74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdfab560308499fb8f607e8fc73b901" is_pre_election: true
I20260619 20:55:03.349458 1242 raft_consensus.cc:2454] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f8b53b6d1446c794ff4d5601baff74 in term 0.
I20260619 20:55:03.349475 1310 leader_election.cc:302] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f8b53b6d1446c794ff4d5601baff74, ce213cf2d4ab4df1ad0d39b97efd4c33; no voters:
I20260619 20:55:03.349613 1449 raft_consensus.cc:2781] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:03.349663 1449 raft_consensus.cc:482] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:03.349686 1449 raft_consensus.cc:3037] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:03.350384 1449 raft_consensus.cc:504] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.350523 1449 leader_election.cc:288] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [CANDIDATE]: Term 1 election: Requested vote from peers dcdfab560308499fb8f607e8fc73b901 (127.0.98.2:42627), ce213cf2d4ab4df1ad0d39b97efd4c33 (127.0.98.1:40399)
I20260619 20:55:03.350642 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55917b48d20c43acb2218f72c83ce713" candidate_uuid: "46f8b53b6d1446c794ff4d5601baff74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdfab560308499fb8f607e8fc73b901"
I20260619 20:55:03.350708 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55917b48d20c43acb2218f72c83ce713" candidate_uuid: "46f8b53b6d1446c794ff4d5601baff74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33"
I20260619 20:55:03.350733 1242 raft_consensus.cc:3037] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:03.350806 1107 raft_consensus.cc:3037] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:03.351420 1242 raft_consensus.cc:2454] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f8b53b6d1446c794ff4d5601baff74 in term 1.
I20260619 20:55:03.351578 1107 raft_consensus.cc:2454] T 55917b48d20c43acb2218f72c83ce713 P ce213cf2d4ab4df1ad0d39b97efd4c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f8b53b6d1446c794ff4d5601baff74 in term 1.
I20260619 20:55:03.351593 1312 leader_election.cc:302] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f8b53b6d1446c794ff4d5601baff74, dcdfab560308499fb8f607e8fc73b901; no voters:
I20260619 20:55:03.351691 1449 raft_consensus.cc:2781] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:03.351843 1449 raft_consensus.cc:686] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [term 1 LEADER]: Becoming Leader. State: Replica: 46f8b53b6d1446c794ff4d5601baff74, State: Running, Role: LEADER
I20260619 20:55:03.351936 1449 consensus_queue.cc:237] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } }
I20260619 20:55:03.352690 961 catalog_manager.cc:5495] T 55917b48d20c43acb2218f72c83ce713 P 46f8b53b6d1446c794ff4d5601baff74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46f8b53b6d1446c794ff4d5601baff74 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "46f8b53b6d1446c794ff4d5601baff74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdfab560308499fb8f607e8fc73b901" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce213cf2d4ab4df1ad0d39b97efd4c33" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f8b53b6d1446c794ff4d5601baff74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43619 } health_report { overall_health: HEALTHY } } }
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:03.361740 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.355159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.0.98.1:40399)
0619 20:55:03.355193 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.355195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.355360 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:55:03.355384 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.355385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.355395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.355454 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.356015 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.356451 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.357094 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.357256 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.357300 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.359480 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.359486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.359487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.359493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.360868 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.361163 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.361165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.361165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.361216 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.361356 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.361356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.361356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.361480 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.361580 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.361629 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.361684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:55:03.364673 1289 tablet.cc:2369] T 55917b48d20c43acb2218f72c83ce713 P dcdfab560308499fb8f607e8fc73b901: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:03.384608 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.377600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.1:40399)
0619 20:55:03.377668 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.377671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.377776 (+ 105us) server_negotiation.cc:407] Connection header received
0619 20:55:03.377867 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.377869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.377879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.377928 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.378526 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.379139 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.379862 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.380071 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.380142 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.382143 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.382144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.382144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.382151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.383403 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.383761 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.383762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.383762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.383814 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.384036 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.384037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.384037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.384132 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.384246 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.384419 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.384495 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:03.404958 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:03.398894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.98.1:40399)
0619 20:55:03.398936 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:03.398938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:03.399010 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:55:03.399061 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:03.399062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:03.399070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:03.399114 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:03.399609 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.400047 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.400716 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:03.400878 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:03.400921 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:03.402992 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:03.402993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:03.402993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:03.402999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:03.404113 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.404388 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.404389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.404389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.404432 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:03.404589 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:03.404589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:03.404590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:03.404666 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:03.404730 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:03.404808 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:55:03.404860 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:55:03.405423 1067 tablet_service.cc:2908] Rejecting scan request for tablet 55917b48d20c43acb2218f72c83ce713: Uninitialized: clean time has not yet been initialized
I20260619 20:55:03.406246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1024
I20260619 20:55:03.411213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1159
I20260619 20:55:03.415555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1294
I20260619 20:55:03.419889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 930
2026-06-19T20:55:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4424 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[1465](info): set up 2 sockets
Jun 19 20:55:03 dist-test-slave-8bgz krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260619 20:55:04.449862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:04 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:04.560741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:04.639773 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:04.640009 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:04.640033 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:04.641165 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:04.641203 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:55:04.641216 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:04.641256 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:04.641283 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:04.641299 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:04.642854 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:04.643095 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:04.643312 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:04.648038 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:04.648080 1478 server_base.cc:1034] running on GCE node
W20260619 20:55:04.648159 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:04.648031 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:04.648450 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:04.648686 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:04.649818 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781902504649808 us; error 27 us; skew 500 ppm
Jun 19 20:55:04 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:04.652149 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:04.652555 1478 webserver.cc:460] Webserver started at http://127.0.98.62:43781/ using document root <none> and password file <none>
I20260619 20:55:04.652738 1478 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:04.652776 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:04.652858 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:04.653792 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "77cbf5ac796f416bb361621292e45e20"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "3f055c2073b57515d0fc5178f78252e1"
server_key_iv: "2f7d3e0304b34f405a3ef99e58b717d1"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.654068 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "77cbf5ac796f416bb361621292e45e20"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "3f055c2073b57515d0fc5178f78252e1"
server_key_iv: "2f7d3e0304b34f405a3ef99e58b717d1"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.655357 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:04.656002 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:04.656176 1478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:04.656250 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "77cbf5ac796f416bb361621292e45e20"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "3f055c2073b57515d0fc5178f78252e1"
server_key_iv: "2f7d3e0304b34f405a3ef99e58b717d1"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.656301 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:04.688061 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:04.689885 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:04.689993 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:04.694514 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33409
I20260619 20:55:04.694571 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33409 every 8 connection(s)
I20260619 20:55:04.694869 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:04.695324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1478
I20260619 20:55:04.695413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:04.695595 392 external_mini_cluster.cc:1428] Setting key 152f760a599f5f3ffad67b52dda878cb
I20260619 20:55:04.695647 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:04.698401 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Bootstrap starting.
I20260619 20:55:04.698982 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:04.699249 1556 log.cc:824] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:04.699892 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: No bootstrap required, opened a new log
Jun 19 20:55:04 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:04.702102 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260619 20:55:04.702286 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:04.702340 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cbf5ac796f416bb361621292e45e20, State: Initialized, Role: FOLLOWER
I20260619 20:55:04.702483 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260619 20:55:04.702593 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:04.702627 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:04.702677 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:04.703267 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260619 20:55:04.703373 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cbf5ac796f416bb361621292e45e20; no voters:
I20260619 20:55:04.703545 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:04.703573 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:04.703682 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [term 1 LEADER]: Becoming Leader. State: Replica: 77cbf5ac796f416bb361621292e45e20, State: Running, Role: LEADER
I20260619 20:55:04.703796 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:04.703830 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260619 20:55:04.704173 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cbf5ac796f416bb361621292e45e20. Latest consensus state: current_term: 1 leader_uuid: "77cbf5ac796f416bb361621292e45e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260619 20:55:04.704185 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cbf5ac796f416bb361621292e45e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbf5ac796f416bb361621292e45e20" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260619 20:55:04.704460 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:04.704370 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:04.705304 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:04.705997 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:04.706039 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:04.696083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.98.62:33409)
0619 20:55:04.696301 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:04.696309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:04.696351 (+ 42us) server_negotiation.cc:407] Connection header received
0619 20:55:04.696723 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:04.696729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:04.696775 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:04.696921 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:04.697515 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.698501 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.699318 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.699713 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.699832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:04.702293 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:04.702298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:04.702298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:04.702310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:04.704231 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.704676 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.704679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.704680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.704812 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.705031 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.705032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.705032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.705342 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:04.705491 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:04.705701 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 20:55:04.705837 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260619 20:55:04.707587 1570 catalog_manager.cc:1347] Generated new cluster ID: 440da671a73640f788a948a34ae1176b
I20260619 20:55:04.707644 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:04.715499 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:04.716207 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:04.732532 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Generated new TSK 0
I20260619 20:55:04.732755 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:04.757175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:04.836324 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:04.836580 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:04.836618 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:04.837827 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:04.837893 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:04.837926 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:04.837965 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:04.839535 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:04.839795 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:04.840013 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:04.845264 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:04.845309 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:04.845451 1582 server_base.cc:1034] running on GCE node
W20260619 20:55:04.845248 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:04.845841 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:04.846072 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:04.847209 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781902504847208 us; error 54 us; skew 500 ppm
Jun 19 20:55:04 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:04.849478 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:04.849937 1582 webserver.cc:460] Webserver started at http://127.0.98.1:41247/ using document root <none> and password file <none>
I20260619 20:55:04.850162 1582 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:04.850247 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:04.850395 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:04.851584 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b127ed249b714b93846035b22367cdc7"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "b1091925f35439c196881743a303592d"
server_key_iv: "706041d27e0eafa7640fc1f50012136d"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.852007 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b127ed249b714b93846035b22367cdc7"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "b1091925f35439c196881743a303592d"
server_key_iv: "706041d27e0eafa7640fc1f50012136d"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.853440 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:04.854148 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:04.854339 1582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:04.854449 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b127ed249b714b93846035b22367cdc7"
format_stamp: "Formatted at 2026-06-19 20:55:04 on dist-test-slave-8bgz"
server_key: "b1091925f35439c196881743a303592d"
server_key_iv: "706041d27e0eafa7640fc1f50012136d"
server_key_version: "encryptionkey@0"
I20260619 20:55:04.854534 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:04.876981 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:04.878901 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:04.879063 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:04.879596 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:04.879642 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:04.879700 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:04.879726 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:55:04 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:04.887254 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42895
I20260619 20:55:04.887328 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42895 every 8 connection(s)
I20260619 20:55:04.887745 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:04.889618 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:04.880088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38677 (local address 127.0.98.62:33409)
0619 20:55:04.880155 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:04.880157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:04.880373 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:55:04.880830 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:04.880831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:04.880863 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:04.880951 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:04.881731 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.882726 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.883738 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.884131 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.884221 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:04.886721 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:04.886723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:04.886723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:04.886731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:04.888374 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.888882 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.888883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.888884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.888939 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.889155 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.889156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.889156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.889251 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:04.889299 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:04.889501 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 20:55:04.889559 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:55:04.889832 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:04.880059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.1:38677)
0619 20:55:04.880229 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:04.880266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:04.880743 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:04.881009 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:04.881012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:04.881353 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:04.881621 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:04.881627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.882839 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.882839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:04.883650 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:04.883652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.884235 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.884235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:04.884390 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:04.884855 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:04.884870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:04.886630 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:04.888444 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:04.888448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:04.888449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:04.888718 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:04.889002 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:04.889003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:04.889003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:04.889099 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:04.889307 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:04.889310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:04.889414 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 20:55:04.889490 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:55:04.889596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260619 20:55:04.890349 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260619 20:55:04.890427 1713 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:04.890594 1713 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260619 20:55:04.891117 1509 ts_manager.cc:194] Registered new tserver with Master: b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895)
I20260619 20:55:04.891805 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38677
I20260619 20:55:04.892046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1582
I20260619 20:55:04.892146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:04.892313 392 external_mini_cluster.cc:1428] Setting key 9b23330fd97e13ebbca23d6989297307
I20260619 20:55:04.897212 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:04.891318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.1:40521)
0619 20:55:04.891436 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:04.891445 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:04.891515 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:04.891631 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:04.891632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:04.891798 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:04.891856 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:04.891877 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.892744 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.892745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:04.893341 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:04.893343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.893696 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.893697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:04.893766 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:04.894168 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:04.894172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:04.894475 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:04.896301 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:04.896302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:04.896302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:04.896471 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:04.896667 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:04.896668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:04.896668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:04.896716 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:04.896986 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:04.896987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:04.897022 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 20:55:04.897076 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:55:04.897146 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260619 20:55:04.897293 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:04.891309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40521 (local address 127.0.98.62:33409)
0619 20:55:04.891353 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:04.891355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:04.891501 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:55:04.891570 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:04.891571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:04.891581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:04.891629 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:04.891969 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.892658 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.893417 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:04.893618 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:04.893688 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:04.894555 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:04.894557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:04.894557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:04.894564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:04.896208 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.896542 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.896544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.896544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.896603 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:04.896777 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:04.896778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:04.896778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:04.896912 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:04.896982 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:04.897119 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 20:55:04.897196 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:04.916812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:04.995317 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:04.995501 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:04.995524 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:04.996697 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:04.996738 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:04.996758 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:04.996806 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:04.998299 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:04.998504 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:04.998719 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:05.003499 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:05.003598 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:05.003541 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:05.003703 1722 server_base.cc:1034] running on GCE node
I20260619 20:55:05.004001 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:05.004194 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:05.005343 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781902505005327 us; error 29 us; skew 500 ppm
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:05.007751 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:05.008122 1722 webserver.cc:460] Webserver started at http://127.0.98.2:35113/ using document root <none> and password file <none>
I20260619 20:55:05.008291 1722 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:05.008332 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:05.008414 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:05.009380 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3dcba4e345848029efb6456d3862b55"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "95e64142e9a0348ee1bb54538761c2f4"
server_key_iv: "db7e74cb9e2dafd5b693a0ddeea7a983"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.009668 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3dcba4e345848029efb6456d3862b55"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "95e64142e9a0348ee1bb54538761c2f4"
server_key_iv: "db7e74cb9e2dafd5b693a0ddeea7a983"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.011041 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:05.011762 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.011996 1722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:05.012066 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "c3dcba4e345848029efb6456d3862b55"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "95e64142e9a0348ee1bb54538761c2f4"
server_key_iv: "db7e74cb9e2dafd5b693a0ddeea7a983"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.012117 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:05.020881 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:05.022817 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:05.022919 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:05.023310 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:05.023342 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.023363 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:05.023424 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.030578 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40861
I20260619 20:55:05.030702 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40861 every 8 connection(s)
I20260619 20:55:05.030997 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:05.031330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1722
I20260619 20:55:05.031418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:05.031584 392 external_mini_cluster.cc:1428] Setting key bfcc6b68c38a1ea4cb917e79ad4be8de
I20260619 20:55:05.035003 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.023916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46097 (local address 127.0.98.62:33409)
0619 20:55:05.023964 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.023966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.024199 (+ 233us) server_negotiation.cc:407] Connection header received
0619 20:55:05.024852 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.024854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.024881 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.024955 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.025881 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.026775 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.028531 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.028863 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.028955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.031893 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.031896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.031896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.031905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.033465 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.033961 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.033962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.033962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.034025 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.034332 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.034333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.034333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.034458 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.034518 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.034859 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.034940 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:05.035243 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.023891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.2:46097)
0619 20:55:05.024056 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.024091 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.024697 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.025062 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.025067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.025459 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:05.025748 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.025755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.026893 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.026894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.028427 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.028430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.028976 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.028977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.029142 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.029769 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:05.029781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:05.031786 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:05.033546 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.033551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.033553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.033871 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.034100 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.034101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.034101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.034255 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.034526 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:05.034531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:05.034687 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.034811 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.034982 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260619 20:55:05.035698 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260619 20:55:05.035785 1853 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:05.035985 1853 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260619 20:55:05.036368 1509 ts_manager.cc:194] Registered new tserver with Master: c3dcba4e345848029efb6456d3862b55 (127.0.98.2:40861)
I20260619 20:55:05.036864 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46097
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 20:55:05.043512 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.036663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.2:34383)
0619 20:55:05.036780 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.036791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.036840 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.037844 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.037845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.038045 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:05.038117 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.038119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.038862 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.038863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.039750 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.039752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.040100 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.040101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.040178 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.040540 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:05.040545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:05.040949 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:05.042476 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.042477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.042478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.042715 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.042885 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.042885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.042886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.042936 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.043227 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:05.043227 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:05.043272 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.043355 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.043425 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260619 20:55:05.043610 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.037019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34383 (local address 127.0.98.62:33409)
0619 20:55:05.037063 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.037070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.037083 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:55:05.037105 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.037107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.037122 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.037186 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.038250 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.038763 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.039844 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.040015 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.040069 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.041028 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.041029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.041030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.041035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.042410 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.042780 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.042781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.042781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.042828 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.043019 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.043021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.043021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.043156 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.043217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.043400 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.043474 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:05.059862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33409
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:05.141359 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:05.141628 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:05.141693 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:05.143074 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:05.143172 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:05.143211 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:05.143254 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:05.144956 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:05.145241 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:05.145481 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:05.150895 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:05.150906 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:05.150907 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:05.151137 1862 server_base.cc:1034] running on GCE node
I20260619 20:55:05.151378 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:05.151607 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:05.152791 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781902505152736 us; error 42 us; skew 500 ppm
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:05.155220 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:05.155616 1862 webserver.cc:460] Webserver started at http://127.0.98.3:45097/ using document root <none> and password file <none>
I20260619 20:55:05.155838 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:05.155911 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:05.156059 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:05.157042 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "04480452c99e48a791e725a1dc0d3eb2"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "60185ae14e8d9e7af0b6084f0528d0fe"
server_key_iv: "40f8f5e5a46d2dc4d78488ce11133fbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.157378 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04480452c99e48a791e725a1dc0d3eb2"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "60185ae14e8d9e7af0b6084f0528d0fe"
server_key_iv: "40f8f5e5a46d2dc4d78488ce11133fbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.158792 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:05.159497 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.159670 1862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:05.159777 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "04480452c99e48a791e725a1dc0d3eb2"
format_stamp: "Formatted at 2026-06-19 20:55:05 on dist-test-slave-8bgz"
server_key: "60185ae14e8d9e7af0b6084f0528d0fe"
server_key_iv: "40f8f5e5a46d2dc4d78488ce11133fbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:05.159855 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:05.193837 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:05.196764 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:05.196954 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:05.197532 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:05.197606 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.197657 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:05.197718 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:05.205453 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45113
I20260619 20:55:05.205608 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45113 every 8 connection(s)
I20260619 20:55:05.205850 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:05.208464 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.198047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60427 (local address 127.0.98.62:33409)
0619 20:55:05.198126 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.198129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.198363 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:55:05.198848 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.198850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.198865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.198962 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.199551 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.200469 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.202181 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.202667 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.202759 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.205455 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.205456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.205457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.205465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.207011 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.207403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.207404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.207405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.207479 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.207861 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.207862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.207862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.207973 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.208034 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.208273 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.208336 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:55:05.208527 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.198027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.3:60427)
0619 20:55:05.198200 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.198267 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.198746 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.199025 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.199029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.199241 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:05.199426 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.199430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.200592 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.200593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.202079 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.202082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.202807 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.202808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.202948 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.203489 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:05.203499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:05.205348 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:05.207083 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.207087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.207089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.207306 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.207604 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.207605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.207606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.207710 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.208039 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:05.208044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:05.208154 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.208239 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.208373 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260619 20:55:05.209093 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33409
I20260619 20:55:05.209173 1993 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:05.209343 1993 heartbeater.cc:511] Master 127.0.98.62:33409 requested a full tablet report, sending...
I20260619 20:55:05.209789 1509 ts_manager.cc:194] Registered new tserver with Master: 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113)
I20260619 20:55:05.210379 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60427
I20260619 20:55:05.214418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 1862
I20260619 20:55:05.214517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:05.214680 392 external_mini_cluster.cc:1428] Setting key 4a3270cb64a7b450da9c22652f02fad4
I20260619 20:55:05.215490 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.210202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44675 (local address 127.0.98.62:33409)
0619 20:55:05.210254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.210256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.210306 (+ 50us) server_negotiation.cc:407] Connection header received
0619 20:55:05.210370 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.210371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.210379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.210426 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.210787 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.211389 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.212076 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.212243 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.212292 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.213134 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.213135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.213135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.213141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.214279 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.214642 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.214643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.214644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.214701 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.214996 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.214997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.214997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.215141 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.215206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.215350 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.215413 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:05.215505 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.210126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.98.3:44675)
0619 20:55:05.210260 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.210269 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.210318 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.210433 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.210433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.210617 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:05.210692 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.210693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.211478 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.211479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.212001 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.212003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.212310 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.212311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.212356 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.212805 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:05.212814 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:05.213059 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:05.214362 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.214363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.214363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.214579 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.214831 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.214832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.214832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.214912 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.215198 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:05.215199 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:05.215240 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.215317 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.215407 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260619 20:55:05.215503 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:05.237488 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.230720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.0.98.62:33409)
0619 20:55:05.230763 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.230765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.230776 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:05.230824 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.230825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.230834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.230883 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.231428 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.231908 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.232707 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.232886 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.232974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.235101 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.235102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.235102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.235108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.236545 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.236854 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.236855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.236855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.236927 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.237097 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.237097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.237097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.237188 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.237256 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.237345 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.237397 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:05.239066 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:05.240649 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:05.246616 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.242823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40861 (local address 127.0.0.1:49390)
0619 20:55:05.243065 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.243102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.243166 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.243511 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.243514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.243516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.243611 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.243613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244534 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.244535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.245500 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.245503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.246286 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.246287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.246363 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.246380 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.246483 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.246537 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":46,"threads_started":1}
I20260619 20:55:05.246636 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.242978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.0.98.2:40861)
0619 20:55:05.243163 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.243168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.243179 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:05.243379 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.243381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.243395 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.243508 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.243691 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244435 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.245598 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.246164 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.246285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.246377 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.246501 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.246542 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260619 20:55:05.247363 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.242827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45113 (local address 127.0.0.1:43744)
0619 20:55:05.243215 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.243224 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.243276 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.243547 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.243548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.243550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.243659 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.243660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244926 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.244926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.246202 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.246203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.247140 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.247141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.247210 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.247215 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.247268 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.247306 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":104,"thread_start_us":71,"threads_started":1}
I20260619 20:55:05.247400 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.242965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.98.3:45113)
0619 20:55:05.243223 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.243227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.243267 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:55:05.243355 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.243357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.243373 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.243477 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.243734 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244823 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.246380 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.247065 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.247135 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.247201 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.247279 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.247315 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":74,"threads_started":1}
I20260619 20:55:05.247697 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 18f48994be5b4157b78e709d88b81757 (DEFAULT_TABLE table=test-table [id=fbc32fe52968409f917042e50c08fa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.248080 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f48994be5b4157b78e709d88b81757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.248167 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 18f48994be5b4157b78e709d88b81757 (DEFAULT_TABLE table=test-table [id=fbc32fe52968409f917042e50c08fa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.248508 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f48994be5b4157b78e709d88b81757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.250545 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.242838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42895 (local address 127.0.0.1:60306)
0619 20:55:05.243268 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.243277 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.243320 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.243547 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.243548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.243564 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.243652 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.243654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244923 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.244923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.246201 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.246203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.250307 (+ 4104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.250309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.250384 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.250389 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.250440 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.250478 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":68,"thread_start_us":83,"threads_started":1}
I20260619 20:55:05.250882 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.243055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.0.98.1:42895)
0619 20:55:05.243184 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.243189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.243336 (+ 147us) server_negotiation.cc:407] Connection header received
0619 20:55:05.243376 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.243378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.243392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.243488 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.243735 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.244824 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.246339 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.247057 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.247130 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.250690 (+ 3560us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.250722 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.250771 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260619 20:55:05.251309 2016 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Bootstrap starting.
I20260619 20:55:05.251452 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 18f48994be5b4157b78e709d88b81757 (DEFAULT_TABLE table=test-table [id=fbc32fe52968409f917042e50c08fa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.251767 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f48994be5b4157b78e709d88b81757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.251945 2016 tablet_bootstrap.cc:654] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.252177 2016 log.cc:824] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:05.252808 2016 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: No bootstrap required, opened a new log
I20260619 20:55:05.252871 2016 ts_tablet_manager.cc:1397] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:05.254325 2016 raft_consensus.cc:348] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.254518 2017 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Bootstrap starting.
I20260619 20:55:05.254570 2016 raft_consensus.cc:374] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.254614 2016 raft_consensus.cc:729] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04480452c99e48a791e725a1dc0d3eb2, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.254742 2016 consensus_queue.cc:260] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.255038 2016 ts_tablet_manager.cc:1428] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:05.255136 2017 tablet_bootstrap.cc:654] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.255165 1993 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260619 20:55:05.255038 2019 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Bootstrap starting.
I20260619 20:55:05.255452 2017 log.cc:824] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:05.256122 2019 tablet_bootstrap.cc:654] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.256167 2017 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: No bootstrap required, opened a new log
I20260619 20:55:05.256240 2017 ts_tablet_manager.cc:1397] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:05.256441 2019 log.cc:824] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:05.257124 2019 tablet_bootstrap.cc:492] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: No bootstrap required, opened a new log
I20260619 20:55:05.257205 2019 ts_tablet_manager.cc:1397] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:05.257936 2017 raft_consensus.cc:348] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.258086 2017 raft_consensus.cc:374] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.258122 2017 raft_consensus.cc:729] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dcba4e345848029efb6456d3862b55, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.258256 2017 consensus_queue.cc:260] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.258505 2017 ts_tablet_manager.cc:1428] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:55:05.258551 1853 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260619 20:55:05.258976 2019 raft_consensus.cc:348] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.259119 2019 raft_consensus.cc:374] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.259157 2019 raft_consensus.cc:729] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b127ed249b714b93846035b22367cdc7, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.259261 2019 consensus_queue.cc:260] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.259541 2019 ts_tablet_manager.cc:1428] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:55:05.259565 1713 heartbeater.cc:503] Master 127.0.98.62:33409 was elected leader, sending a full tablet report...
I20260619 20:55:05.264691 2023 raft_consensus.cc:482] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:05.264753 2023 raft_consensus.cc:504] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.265023 2023 leader_election.cc:288] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113), b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895)
I20260619 20:55:05.267812 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.265274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39265 (local address 127.0.98.3:45113)
0619 20:55:05.265314 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.265316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.265325 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:05.265370 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.265371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.265381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.265434 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.265563 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.266035 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267060 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.267425 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.267549 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.267719 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.267751 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:05.267825 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.265166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45113 (local address 127.0.98.2:39265)
0619 20:55:05.265263 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.265274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.265316 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.265437 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.265438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.265440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.265492 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.265494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.266127 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.266128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.266946 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.266948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.267539 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.267620 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.267624 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.267711 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.267751 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3}
I20260619 20:55:05.267874 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.265253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35801 (local address 127.0.98.1:42895)
0619 20:55:05.265329 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.265337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.265442 (+ 105us) server_negotiation.cc:407] Connection header received
0619 20:55:05.265503 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.265504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.265514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.265567 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.265718 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.266177 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267151 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.267527 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.267632 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.267779 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.267812 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 20:55:05.267923 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.265195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42895 (local address 127.0.98.2:35801)
0619 20:55:05.265374 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.265382 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.265455 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.265570 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.265571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.265583 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.265651 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.265652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.266301 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.266302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.267047 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.267049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.267628 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.267629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.267703 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.267706 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.267774 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.267810 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260619 20:55:05.268231 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f48994be5b4157b78e709d88b81757" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7" is_pre_election: true
I20260619 20:55:05.268316 1667 raft_consensus.cc:2454] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 0.
I20260619 20:55:05.268523 1740 leader_election.cc:302] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b127ed249b714b93846035b22367cdc7, c3dcba4e345848029efb6456d3862b55; no voters:
I20260619 20:55:05.268635 2023 raft_consensus.cc:2781] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:05.268646 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f48994be5b4157b78e709d88b81757" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04480452c99e48a791e725a1dc0d3eb2" is_pre_election: true
I20260619 20:55:05.268693 2023 raft_consensus.cc:482] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:05.268715 2023 raft_consensus.cc:3037] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.268746 1947 raft_consensus.cc:2454] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 0.
I20260619 20:55:05.269582 2023 raft_consensus.cc:504] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.269678 2023 leader_election.cc:288] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 election: Requested vote from peers 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113), b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895)
I20260619 20:55:05.269836 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f48994be5b4157b78e709d88b81757" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7"
I20260619 20:55:05.269908 1667 raft_consensus.cc:3037] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.270627 1667 raft_consensus.cc:2454] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 1.
I20260619 20:55:05.270748 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f48994be5b4157b78e709d88b81757" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04480452c99e48a791e725a1dc0d3eb2"
I20260619 20:55:05.270824 1947 raft_consensus.cc:3037] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.270812 1740 leader_election.cc:302] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b127ed249b714b93846035b22367cdc7, c3dcba4e345848029efb6456d3862b55; no voters:
I20260619 20:55:05.271010 2023 raft_consensus.cc:2781] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:05.271152 2023 raft_consensus.cc:686] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 1 LEADER]: Becoming Leader. State: Replica: c3dcba4e345848029efb6456d3862b55, State: Running, Role: LEADER
I20260619 20:55:05.271241 2023 consensus_queue.cc:237] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.271647 1947 raft_consensus.cc:2454] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 1.
I20260619 20:55:05.271883 1509 catalog_manager.cc:5495] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dcba4e345848029efb6456d3862b55 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c3dcba4e345848029efb6456d3862b55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } health_report { overall_health: UNKNOWN } } }
W20260619 20:55:05.282140 1854 tablet.cc:2369] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:05.283280 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.280882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.2:40861)
0619 20:55:05.280951 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.280953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.280963 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:05.280994 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.280995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.281019 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.281090 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:05.281210 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.281681 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.282560 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.282774 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.282839 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.282997 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.283079 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.283141 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.283198 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.283214 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:55:05.284839 1667 raft_consensus.cc:1261] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Refusing update from remote peer c3dcba4e345848029efb6456d3862b55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.284853 1947 raft_consensus.cc:1261] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Refusing update from remote peer c3dcba4e345848029efb6456d3862b55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.285153 2026 consensus_queue.cc:1035] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.285257 2023 consensus_queue.cc:1035] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.287638 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298672661644070912 to 7298672661591257088. Current Snapshot: MvccSnapshot[applied={T|T < 7298672661644070912}]
I20260619 20:55:05.288285 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298672661644070912 to 7298672661591257088. Current Snapshot: MvccSnapshot[applied={T|T < 7298672661644070912}]
I20260619 20:55:05.289150 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298672661644070912 to 7298672661591257088. Current Snapshot: MvccSnapshot[applied={T|T < 7298672661644070912 or (T in {7298672661644070912})}]
I20260619 20:55:05.290918 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:05.291003 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:05.291675 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Sending DeleteTablet for 3 replicas of tablet 18f48994be5b4157b78e709d88b81757
I20260619 20:55:05.291992 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f48994be5b4157b78e709d88b81757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49390
I20260619 20:55:05.291990 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f48994be5b4157b78e709d88b81757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43744
I20260619 20:55:05.292052 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f48994be5b4157b78e709d88b81757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60306
I20260619 20:55:05.292192 2039 tablet_replica.cc:330] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: stopping tablet replica
I20260619 20:55:05.292284 2039 raft_consensus.cc:2229] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:05.292284 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:05.292424 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:05.292630 2039 raft_consensus.cc:2258] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.293167 2039 ts_tablet_manager.cc:1905] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.293567 1787 tablet_service.cc:1467] Processing CreateTablet for tablet fc182417e2dd42d8a3abdcf087563af8 (DEFAULT_TABLE table=test-table [id=8419ef2036334cc4aad0e7c55da528b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.293608 1647 tablet_service.cc:1467] Processing CreateTablet for tablet fc182417e2dd42d8a3abdcf087563af8 (DEFAULT_TABLE table=test-table [id=8419ef2036334cc4aad0e7c55da528b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.293608 1927 tablet_service.cc:1467] Processing CreateTablet for tablet fc182417e2dd42d8a3abdcf087563af8 (DEFAULT_TABLE table=test-table [id=8419ef2036334cc4aad0e7c55da528b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:05.293673 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc182417e2dd42d8a3abdcf087563af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.293709 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc182417e2dd42d8a3abdcf087563af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.293709 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc182417e2dd42d8a3abdcf087563af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.294981 2017 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Bootstrap starting.
I20260619 20:55:05.295352 2019 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Bootstrap starting.
I20260619 20:55:05.295375 2017 tablet_bootstrap.cc:654] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.295697 2016 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Bootstrap starting.
I20260619 20:55:05.295763 2019 tablet_bootstrap.cc:654] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.296095 2016 tablet_bootstrap.cc:654] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.296484 2017 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: No bootstrap required, opened a new log
I20260619 20:55:05.296526 2017 ts_tablet_manager.cc:1397] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:55:05.296674 2017 raft_consensus.cc:348] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.296748 2017 raft_consensus.cc:374] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.296775 2017 raft_consensus.cc:729] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dcba4e345848029efb6456d3862b55, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.296823 2017 consensus_queue.cc:260] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.296922 2017 ts_tablet_manager.cc:1428] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:05.297000 2019 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: No bootstrap required, opened a new log
I20260619 20:55:05.297045 2019 ts_tablet_manager.cc:1397] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:05.297178 2019 raft_consensus.cc:348] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.297256 2019 raft_consensus.cc:374] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.297276 2019 raft_consensus.cc:729] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b127ed249b714b93846035b22367cdc7, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.297309 2019 consensus_queue.cc:260] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.297396 2019 ts_tablet_manager.cc:1428] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:05.298013 2039 ts_tablet_manager.cc:1918] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:05.298085 2039 log.cc:1196] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/18f48994be5b4157b78e709d88b81757
I20260619 20:55:05.298280 2038 tablet_replica.cc:330] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: stopping tablet replica
I20260619 20:55:05.298374 2039 ts_tablet_manager.cc:1939] T 18f48994be5b4157b78e709d88b81757 P 04480452c99e48a791e725a1dc0d3eb2: Deleting consensus metadata
I20260619 20:55:05.298384 2038 raft_consensus.cc:2229] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:05.298492 2038 raft_consensus.cc:2258] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.298815 1494 catalog_manager.cc:4841] TS 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113): tablet 18f48994be5b4157b78e709d88b81757 (table test-table [id=fbc32fe52968409f917042e50c08fa89]) successfully deleted
I20260619 20:55:05.298913 2016 tablet_bootstrap.cc:492] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: No bootstrap required, opened a new log
I20260619 20:55:05.298981 2016 ts_tablet_manager.cc:1397] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:55:05.299109 2016 raft_consensus.cc:348] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.299175 2016 raft_consensus.cc:374] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.299202 2016 raft_consensus.cc:729] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04480452c99e48a791e725a1dc0d3eb2, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.299247 2016 consensus_queue.cc:260] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.299327 2016 ts_tablet_manager.cc:1428] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:05.299744 2038 ts_tablet_manager.cc:1905] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.301142 2038 ts_tablet_manager.cc:1918] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:05.301213 2038 log.cc:1196] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/18f48994be5b4157b78e709d88b81757
I20260619 20:55:05.301435 2038 ts_tablet_manager.cc:1939] T 18f48994be5b4157b78e709d88b81757 P c3dcba4e345848029efb6456d3862b55: Deleting consensus metadata
I20260619 20:55:05.301838 1497 catalog_manager.cc:4841] TS c3dcba4e345848029efb6456d3862b55 (127.0.98.2:40861): tablet 18f48994be5b4157b78e709d88b81757 (table test-table [id=fbc32fe52968409f917042e50c08fa89]) successfully deleted
I20260619 20:55:05.302361 2040 tablet_replica.cc:330] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: stopping tablet replica
I20260619 20:55:05.302520 2040 raft_consensus.cc:2229] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:05.302635 2040 raft_consensus.cc:2258] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.302991 2040 ts_tablet_manager.cc:1905] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.304384 2040 ts_tablet_manager.cc:1918] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:05.304486 2040 log.cc:1196] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/18f48994be5b4157b78e709d88b81757
I20260619 20:55:05.304734 2040 ts_tablet_manager.cc:1939] T 18f48994be5b4157b78e709d88b81757 P b127ed249b714b93846035b22367cdc7: Deleting consensus metadata
I20260619 20:55:05.305133 1496 catalog_manager.cc:4841] TS b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895): tablet 18f48994be5b4157b78e709d88b81757 (table test-table [id=fbc32fe52968409f917042e50c08fa89]) successfully deleted
W20260619 20:55:05.388484 1714 tablet.cc:2369] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:05.397699 2026 raft_consensus.cc:482] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:05.397791 2026 raft_consensus.cc:504] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.397967 2026 leader_election.cc:288] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895), 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113)
I20260619 20:55:05.398272 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc182417e2dd42d8a3abdcf087563af8" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04480452c99e48a791e725a1dc0d3eb2" is_pre_election: true
I20260619 20:55:05.398283 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc182417e2dd42d8a3abdcf087563af8" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7" is_pre_election: true
I20260619 20:55:05.398404 1947 raft_consensus.cc:2454] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 0.
I20260619 20:55:05.398415 1667 raft_consensus.cc:2454] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 0.
I20260619 20:55:05.398568 1738 leader_election.cc:302] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04480452c99e48a791e725a1dc0d3eb2, c3dcba4e345848029efb6456d3862b55; no voters:
I20260619 20:55:05.398690 2026 raft_consensus.cc:2781] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:05.398722 2026 raft_consensus.cc:482] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:05.398742 2026 raft_consensus.cc:3037] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.399379 2026 raft_consensus.cc:504] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.399514 2026 leader_election.cc:288] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 election: Requested vote from peers b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895), 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113)
I20260619 20:55:05.399660 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc182417e2dd42d8a3abdcf087563af8" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7"
I20260619 20:55:05.399693 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc182417e2dd42d8a3abdcf087563af8" candidate_uuid: "c3dcba4e345848029efb6456d3862b55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04480452c99e48a791e725a1dc0d3eb2"
I20260619 20:55:05.399740 1667 raft_consensus.cc:3037] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.399768 1947 raft_consensus.cc:3037] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.400422 1947 raft_consensus.cc:2454] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 1.
I20260619 20:55:05.400520 1667 raft_consensus.cc:2454] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcba4e345848029efb6456d3862b55 in term 1.
I20260619 20:55:05.400591 1738 leader_election.cc:302] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04480452c99e48a791e725a1dc0d3eb2, c3dcba4e345848029efb6456d3862b55; no voters:
I20260619 20:55:05.400665 2026 raft_consensus.cc:2781] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:05.400708 2026 raft_consensus.cc:686] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 1 LEADER]: Becoming Leader. State: Replica: c3dcba4e345848029efb6456d3862b55, State: Running, Role: LEADER
I20260619 20:55:05.400751 2026 consensus_queue.cc:237] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } }
I20260619 20:55:05.401262 1510 catalog_manager.cc:5495] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dcba4e345848029efb6456d3862b55 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c3dcba4e345848029efb6456d3862b55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:05.441431 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.434798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.0.1:55414)
0619 20:55:05.434936 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.434948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.435020 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.435151 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.435152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.435504 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:05.435562 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.435564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.436200 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.436200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.436772 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.436774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.437076 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.437076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.437122 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.437588 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:05.437593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:05.438946 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:05.440510 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.440515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.440517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.440684 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.440858 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:05.440858 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:05.440858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:05.440908 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:05.441168 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:05.441173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:05.441211 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.441265 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.441345 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260619 20:55:05.441501 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.434890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.98.62:33409)
0619 20:55:05.434972 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.434974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.435004 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:55:05.435072 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.435073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.435081 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.435126 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.435630 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.436125 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.436840 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.437010 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.437062 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.439024 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.439025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.439026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.439031 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.440439 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.440758 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.440759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.440759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.440805 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.440997 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.440997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.440998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.441089 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.441177 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.441291 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.441387 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":65}
I20260619 20:55:05.443305 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55414:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:55:05.443476 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:05.445044 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 39592ee106c7417ea6779dc3c8acc816 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b7655ed72e64203a8b691d489441e09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:05.445052 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 39592ee106c7417ea6779dc3c8acc816 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b7655ed72e64203a8b691d489441e09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:05.445067 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 39592ee106c7417ea6779dc3c8acc816 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b7655ed72e64203a8b691d489441e09]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:05.445202 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39592ee106c7417ea6779dc3c8acc816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.445204 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39592ee106c7417ea6779dc3c8acc816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.445222 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39592ee106c7417ea6779dc3c8acc816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:05.446355 2019 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: Bootstrap starting.
I20260619 20:55:05.446671 2017 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: Bootstrap starting.
I20260619 20:55:05.446769 2019 tablet_bootstrap.cc:654] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.447005 2017 tablet_bootstrap.cc:654] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.447685 2019 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: No bootstrap required, opened a new log
I20260619 20:55:05.447706 2016 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: Bootstrap starting.
I20260619 20:55:05.447739 2019 ts_tablet_manager.cc:1397] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:05.447883 2019 raft_consensus.cc:348] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.447962 2019 raft_consensus.cc:374] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.447983 2019 raft_consensus.cc:729] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b127ed249b714b93846035b22367cdc7, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.448031 2019 consensus_queue.cc:260] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.448094 2016 tablet_bootstrap.cc:654] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:05.448153 2019 ts_tablet_manager.cc:1428] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:05.448199 2024 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.448320 2024 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.448622 2017 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: No bootstrap required, opened a new log
I20260619 20:55:05.448678 2017 ts_tablet_manager.cc:1397] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:55:05.448783 2017 raft_consensus.cc:348] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.448848 2017 raft_consensus.cc:374] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.448861 2017 raft_consensus.cc:729] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dcba4e345848029efb6456d3862b55, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.448890 2017 consensus_queue.cc:260] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.448997 2017 ts_tablet_manager.cc:1428] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:05.449013 2026 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.449093 2026 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.449828 2016 tablet_bootstrap.cc:492] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: No bootstrap required, opened a new log
I20260619 20:55:05.449882 2016 ts_tablet_manager.cc:1397] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:05.450021 2016 raft_consensus.cc:348] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.450098 2016 raft_consensus.cc:374] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:05.450119 2016 raft_consensus.cc:729] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04480452c99e48a791e725a1dc0d3eb2, State: Initialized, Role: FOLLOWER
I20260619 20:55:05.450160 2016 consensus_queue.cc:260] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.450289 2016 ts_tablet_manager.cc:1428] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:05.450345 2020 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.450460 2020 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 20:55:05.456339 1994 tablet.cc:2369] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:05.602874 2020 raft_consensus.cc:482] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:05.603037 2020 raft_consensus.cc:504] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.603515 2020 leader_election.cc:288] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3dcba4e345848029efb6456d3862b55 (127.0.98.2:40861), b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895)
I20260619 20:55:05.606980 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.603754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34609 (local address 127.0.98.1:42895)
0619 20:55:05.603804 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.603806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.603816 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:05.603857 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.603857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.603892 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.603950 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.604220 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.604726 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606151 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.606585 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606638 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.606646 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.606861 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.606911 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:55:05.606981 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.603677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42895 (local address 127.0.98.3:34609)
0619 20:55:05.603732 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.603745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.603792 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.603993 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.603995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.603997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.604123 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.604126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.604809 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.604810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.606050 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.606053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.606662 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.606755 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.606758 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.606829 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.606869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260619 20:55:05.607038 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.603846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53315 (local address 127.0.98.2:40861)
0619 20:55:05.603944 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.603946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.604096 (+ 150us) server_negotiation.cc:407] Connection header received
0619 20:55:05.604130 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.604131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.604142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.604228 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:05.604428 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.605095 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606196 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.606609 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606707 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.606734 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.606921 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.606964 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 20:55:05.607086 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.603678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40861 (local address 127.0.98.3:53315)
0619 20:55:05.603991 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.604004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.604081 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.604279 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.604280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.604282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:05.604349 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.604350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.605193 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.605194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.606063 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.606066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.606714 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.606715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.606786 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.606841 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.606899 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.606931 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":161,"thread_start_us":103,"threads_started":1}
I20260619 20:55:05.607312 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39592ee106c7417ea6779dc3c8acc816" candidate_uuid: "04480452c99e48a791e725a1dc0d3eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7" is_pre_election: true
I20260619 20:55:05.607419 1667 raft_consensus.cc:2454] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04480452c99e48a791e725a1dc0d3eb2 in term 0.
I20260619 20:55:05.607499 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39592ee106c7417ea6779dc3c8acc816" candidate_uuid: "04480452c99e48a791e725a1dc0d3eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dcba4e345848029efb6456d3862b55" is_pre_election: true
I20260619 20:55:05.607611 1880 leader_election.cc:302] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04480452c99e48a791e725a1dc0d3eb2, b127ed249b714b93846035b22367cdc7; no voters:
I20260619 20:55:05.607674 1807 raft_consensus.cc:2454] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04480452c99e48a791e725a1dc0d3eb2 in term 0.
I20260619 20:55:05.607741 2020 raft_consensus.cc:2781] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:05.607779 2020 raft_consensus.cc:482] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:05.607795 2020 raft_consensus.cc:3037] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.608534 2020 raft_consensus.cc:504] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.608673 2020 leader_election.cc:288] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c3dcba4e345848029efb6456d3862b55 (127.0.98.2:40861), b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895)
I20260619 20:55:05.608829 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39592ee106c7417ea6779dc3c8acc816" candidate_uuid: "04480452c99e48a791e725a1dc0d3eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127ed249b714b93846035b22367cdc7"
I20260619 20:55:05.608794 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39592ee106c7417ea6779dc3c8acc816" candidate_uuid: "04480452c99e48a791e725a1dc0d3eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dcba4e345848029efb6456d3862b55"
I20260619 20:55:05.608930 1807 raft_consensus.cc:3037] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.608930 1667 raft_consensus.cc:3037] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:05.609562 1667 raft_consensus.cc:2454] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04480452c99e48a791e725a1dc0d3eb2 in term 1.
I20260619 20:55:05.609722 1807 raft_consensus.cc:2454] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04480452c99e48a791e725a1dc0d3eb2 in term 1.
I20260619 20:55:05.609803 1880 leader_election.cc:302] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04480452c99e48a791e725a1dc0d3eb2, b127ed249b714b93846035b22367cdc7; no voters:
I20260619 20:55:05.609897 2020 raft_consensus.cc:2781] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:05.610031 2020 raft_consensus.cc:686] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 04480452c99e48a791e725a1dc0d3eb2, State: Running, Role: LEADER
I20260619 20:55:05.610101 2020 consensus_queue.cc:237] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } }
I20260619 20:55:05.610430 2055 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: TxnStatusTablet state changed. Reason: New leader 04480452c99e48a791e725a1dc0d3eb2. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.610527 2055 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:05.610677 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:55:05.610805 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:55:05.610854 1510 catalog_manager.cc:5495] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:05.632009 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.629201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.98.3:45113)
0619 20:55:05.629239 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.629241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.629274 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:55:05.629358 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.629359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.629373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.629425 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:05.629545 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.630124 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.631278 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.631452 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.631507 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.631689 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.631772 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.631823 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.631934 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.631951 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:05.632048 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.629152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45113 (local address 127.0.0.1:43754)
0619 20:55:05.629227 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.629238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.629286 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.629437 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.629438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.629440 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:05.629485 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.629488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.630235 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.630236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.631206 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.631208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.631526 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.631527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.631590 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.631605 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.631866 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.631873 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.631924 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.631969 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:55:05.633147 1667 raft_consensus.cc:1261] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Refusing update from remote peer 04480452c99e48a791e725a1dc0d3eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.633261 1807 raft_consensus.cc:1261] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Refusing update from remote peer 04480452c99e48a791e725a1dc0d3eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.633373 2055 consensus_queue.cc:1035] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.633622 2055 consensus_queue.cc:1035] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.633936 2024 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: TxnStatusTablet state changed. Reason: New leader 04480452c99e48a791e725a1dc0d3eb2. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.634042 2024 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.634629 2026 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: TxnStatusTablet state changed. Reason: New leader 04480452c99e48a791e725a1dc0d3eb2. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.634724 2026 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.635250 2062 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.635344 2062 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:05.635411 2020 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.635474 2020 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P 04480452c99e48a791e725a1dc0d3eb2: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:05.635766 2024 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.635864 2024 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P b127ed249b714b93846035b22367cdc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.636497 2026 tablet_replica.cc:439] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04480452c99e48a791e725a1dc0d3eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dcba4e345848029efb6456d3862b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40861 } } peers { permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 } } peers { permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 } } }
I20260619 20:55:05.636569 2026 tablet_replica.cc:442] T 39592ee106c7417ea6779dc3c8acc816 P c3dcba4e345848029efb6456d3862b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:05.641525 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.639108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34623 (local address 127.0.98.3:45113)
0619 20:55:05.639141 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.639143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.639205 (+ 62us) server_negotiation.cc:407] Connection header received
0619 20:55:05.639265 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.639266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.639273 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.639316 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:05.639421 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.639914 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.641010 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.641161 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.641206 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.641366 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.641387 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.641419 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.641472 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.641485 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:05.641748 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.639038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45113 (local address 127.0.98.2:34623)
0619 20:55:05.639149 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.639158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.639211 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.639319 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.639320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.639322 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:05.639365 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.639367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.639991 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.639991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.640929 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.640931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.641228 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.641229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.641283 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.641297 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.641422 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.641423 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.641468 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.641509 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":46,"threads_started":1}
I20260619 20:55:05.643882 1667 raft_consensus.cc:1261] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Refusing update from remote peer c3dcba4e345848029efb6456d3862b55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.643896 1947 raft_consensus.cc:1261] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Refusing update from remote peer c3dcba4e345848029efb6456d3862b55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:05.644058 2023 consensus_queue.cc:1035] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04480452c99e48a791e725a1dc0d3eb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.644131 2026 consensus_queue.cc:1035] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b127ed249b714b93846035b22367cdc7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:05.651798 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.649267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44083 (local address 127.0.98.2:40861)
0619 20:55:05.649301 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.649303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.649310 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:05.649330 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.649330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.649340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.649400 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:05.649479 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.650185 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.651141 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.651357 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.651439 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.651603 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.651631 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.651673 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.651722 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.651740 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:05.651810 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.649186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40861 (local address 127.0.98.3:44083)
0619 20:55:05.649237 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:05.649247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:05.649280 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:05.649399 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:05.649400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:05.649400 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:05.649443 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.649444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.650273 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.650274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.651036 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:05.651038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.651440 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.651441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:05.651497 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.651509 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:05.651672 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:05.651674 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:05.651719 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:55:05.651753 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 20:55:05.660300 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:05.660401 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:05.661005 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77cbf5ac796f416bb361621292e45e20: Sending DeleteTablet for 3 replicas of tablet fc182417e2dd42d8a3abdcf087563af8
I20260619 20:55:05.661262 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet fc182417e2dd42d8a3abdcf087563af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43744
I20260619 20:55:05.661293 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet fc182417e2dd42d8a3abdcf087563af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49390
I20260619 20:55:05.661342 2039 tablet_replica.cc:330] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: stopping tablet replica
I20260619 20:55:05.661381 2039 raft_consensus.cc:2229] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:05.661408 2039 raft_consensus.cc:2258] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.661401 2038 tablet_replica.cc:330] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: stopping tablet replica
I20260619 20:55:05.661396 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet fc182417e2dd42d8a3abdcf087563af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60306
I20260619 20:55:05.661458 2038 raft_consensus.cc:2229] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:05.661468 2040 tablet_replica.cc:330] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: stopping tablet replica
I20260619 20:55:05.661500 2038 raft_consensus.cc:2258] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.661509 2040 raft_consensus.cc:2229] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:05.661542 2040 raft_consensus.cc:2258] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:05.661831 2040 ts_tablet_manager.cc:1905] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.661862 2038 ts_tablet_manager.cc:1905] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.663745 2038 ts_tablet_manager.cc:1918] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:05.663833 2038 log.cc:1196] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/fc182417e2dd42d8a3abdcf087563af8
I20260619 20:55:05.663872 2039 ts_tablet_manager.cc:1905] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:05.664054 2038 ts_tablet_manager.cc:1939] T fc182417e2dd42d8a3abdcf087563af8 P c3dcba4e345848029efb6456d3862b55: Deleting consensus metadata
I20260619 20:55:05.664372 1497 catalog_manager.cc:4841] TS c3dcba4e345848029efb6456d3862b55 (127.0.98.2:40861): tablet fc182417e2dd42d8a3abdcf087563af8 (table test-table [id=8419ef2036334cc4aad0e7c55da528b2]) successfully deleted
I20260619 20:55:05.664497 2040 ts_tablet_manager.cc:1918] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:05.664556 2040 log.cc:1196] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/fc182417e2dd42d8a3abdcf087563af8
I20260619 20:55:05.664757 2040 ts_tablet_manager.cc:1939] T fc182417e2dd42d8a3abdcf087563af8 P b127ed249b714b93846035b22367cdc7: Deleting consensus metadata
I20260619 20:55:05.665076 1496 catalog_manager.cc:4841] TS b127ed249b714b93846035b22367cdc7 (127.0.98.1:42895): tablet fc182417e2dd42d8a3abdcf087563af8 (table test-table [id=8419ef2036334cc4aad0e7c55da528b2]) successfully deleted
I20260619 20:55:05.665887 2039 ts_tablet_manager.cc:1918] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:05.665940 2039 log.cc:1196] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/fc182417e2dd42d8a3abdcf087563af8
I20260619 20:55:05.666158 2039 ts_tablet_manager.cc:1939] T fc182417e2dd42d8a3abdcf087563af8 P 04480452c99e48a791e725a1dc0d3eb2: Deleting consensus metadata
I20260619 20:55:05.666477 1494 catalog_manager.cc:4841] TS 04480452c99e48a791e725a1dc0d3eb2 (127.0.98.3:45113): tablet fc182417e2dd42d8a3abdcf087563af8 (table test-table [id=8419ef2036334cc4aad0e7c55da528b2]) successfully deleted
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902505, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:05.673770 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.663835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.0.98.1:42895)
0619 20:55:05.664275 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.664277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.665807 (+ 1530us) server_negotiation.cc:407] Connection header received
0619 20:55:05.665838 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.665839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.665851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.665909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.666621 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.667097 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.667834 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.668047 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.668127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.670930 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.670938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.670938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.670946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.672602 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.672979 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.672983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.672983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.673041 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.673248 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.673250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.673250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.673383 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.673553 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.673597 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.673673 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
W20260619 20:55:05.674011 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60322
I20260619 20:55:05.680774 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:05.674914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.0.98.62:33409)
0619 20:55:05.674963 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:05.674965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:05.674972 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:05.675035 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:05.675035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:05.675045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:05.675100 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:05.675582 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.676062 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.676742 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:05.676952 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:05.677047 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:05.678200 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:05.678201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:05.678202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:05.678241 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:05.679675 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.680025 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.680026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.680026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.680078 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:05.680289 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:05.680290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:05.680290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:05.680402 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:05.680511 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:05.680611 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:55:05.680681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 20:55:05.681087 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55428
I20260619 20:55:05.681720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1582
I20260619 20:55:05.687681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1722
I20260619 20:55:05.694589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1862
I20260619 20:55:05.702100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 1478
2026-06-19T20:55:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2282 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[2078](info): set up 2 sockets
Jun 19 20:55:05 dist-test-slave-8bgz krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260619 20:55:06.741922 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:06 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:06.858981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:06.938423 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:06.938619 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:06.938642 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:06.939749 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:06.939785 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:55:06.939798 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:06.939811 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:06.939824 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:06.939836 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:06.941206 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:06.941392 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:06.941551 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:06.946591 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:06.946625 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:06.946630 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:06.946779 2091 server_base.cc:1034] running on GCE node
I20260619 20:55:06.947118 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:06.947479 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:06.948673 2091 hybrid_clock.cc:648] HybridClock initialized: now 1781902506948644 us; error 39 us; skew 500 ppm
Jun 19 20:55:06 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:06.951711 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:06.952167 2091 webserver.cc:460] Webserver started at http://127.0.98.62:36455/ using document root <none> and password file <none>
I20260619 20:55:06.952411 2091 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:06.952483 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:06.952617 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:06.953714 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "deaefb012cf846609931edb8a5959b14"
format_stamp: "Formatted at 2026-06-19 20:55:06 on dist-test-slave-8bgz"
server_key: "71c04f0a23541f35f3ae54444270462a"
server_key_iv: "8cb8f35e77895bb69b88221632a788cd"
server_key_version: "encryptionkey@0"
I20260619 20:55:06.954085 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "deaefb012cf846609931edb8a5959b14"
format_stamp: "Formatted at 2026-06-19 20:55:06 on dist-test-slave-8bgz"
server_key: "71c04f0a23541f35f3ae54444270462a"
server_key_iv: "8cb8f35e77895bb69b88221632a788cd"
server_key_version: "encryptionkey@0"
I20260619 20:55:06.955425 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:06.956209 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:06.956382 2091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:06.956481 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "deaefb012cf846609931edb8a5959b14"
format_stamp: "Formatted at 2026-06-19 20:55:06 on dist-test-slave-8bgz"
server_key: "71c04f0a23541f35f3ae54444270462a"
server_key_iv: "8cb8f35e77895bb69b88221632a788cd"
server_key_version: "encryptionkey@0"
I20260619 20:55:06.956562 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:06.970916 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:06.973177 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:06.973340 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:06.977658 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35449
I20260619 20:55:06.977761 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35449 every 8 connection(s)
I20260619 20:55:06.978050 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:06.978806 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:06.980934 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Bootstrap starting.
I20260619 20:55:06.981503 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:06.981793 2169 log.cc:824] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:06.982467 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: No bootstrap required, opened a new log
I20260619 20:55:06.983728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2091
I20260619 20:55:06.983853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:06.984014 392 external_mini_cluster.cc:1428] Setting key 5bea6520097e351fd9847e6e685a6c00
I20260619 20:55:06.984741 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260619 20:55:06.984905 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:06.984951 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deaefb012cf846609931edb8a5959b14, State: Initialized, Role: FOLLOWER
I20260619 20:55:06.985105 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260619 20:55:06.985205 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:06.985258 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:06.985311 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:06.985787 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260619 20:55:06.985906 2169 leader_election.cc:302] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deaefb012cf846609931edb8a5959b14; no voters:
I20260619 20:55:06.986083 2169 leader_election.cc:288] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:06.986150 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:06.986369 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [term 1 LEADER]: Becoming Leader. State: Replica: deaefb012cf846609931edb8a5959b14, State: Running, Role: LEADER
I20260619 20:55:06.986461 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260619 20:55:06.986359 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:06.986855 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deaefb012cf846609931edb8a5959b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } } }
I20260619 20:55:06.986931 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:06.986894 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deaefb012cf846609931edb8a5959b14. Latest consensus state: current_term: 1 leader_uuid: "deaefb012cf846609931edb8a5959b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deaefb012cf846609931edb8a5959b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } } }
I20260619 20:55:06.986972 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:06.987133 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:06.987593 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:06.990379 2178 catalog_manager.cc:1347] Generated new cluster ID: 948cba5af5994a82bd76ebfb8b50fc15
I20260619 20:55:06.990419 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:55:06 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:06.995082 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:06.984549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.98.62:35449)
0619 20:55:06.984730 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:06.984737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:06.984760 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:55:06.984857 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:06.984862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:06.984900 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:06.985048 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:06.985749 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:06.987008 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:06.988095 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:06.988706 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:06.988843 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:06.991507 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:06.991511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:06.991511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:06.991522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:06.993437 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:06.993813 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:06.993817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:06.993818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:06.993984 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:06.994195 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:06.994196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:06.994197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:06.994510 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:06.994669 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:06.994754 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:55:06.994887 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260619 20:55:07.019961 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:07.020727 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:07.032639 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Generated new TSK 0
I20260619 20:55:07.032859 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:07.065336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:07.144191 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:07.144377 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:07.144402 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:07.145650 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:07.145692 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:07.145713 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:55:07.145725 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:07.145790 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:07.147267 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:07.147531 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:07.147760 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:07.152858 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.152881 2195 server_base.cc:1034] running on GCE node
W20260619 20:55:07.152895 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:07.152869 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.153218 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:07.153429 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:07.154597 2195 hybrid_clock.cc:648] HybridClock initialized: now 1781902507154573 us; error 38 us; skew 500 ppm
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:07.156890 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:07.157351 2195 webserver.cc:460] Webserver started at http://127.0.98.1:46297/ using document root <none> and password file <none>
I20260619 20:55:07.157586 2195 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:07.157660 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:07.157809 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:07.158911 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b6621e97316e4b4084545187b8f1310a"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "fce077924afc388d0161039473c70396"
server_key_iv: "faa56ace26c2574375cc5eb80fb7786a"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.159340 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b6621e97316e4b4084545187b8f1310a"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "fce077924afc388d0161039473c70396"
server_key_iv: "faa56ace26c2574375cc5eb80fb7786a"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.160712 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:07.161429 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.161614 2195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:07.161712 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b6621e97316e4b4084545187b8f1310a"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "fce077924afc388d0161039473c70396"
server_key_iv: "faa56ace26c2574375cc5eb80fb7786a"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.161810 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:07.175658 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:07.177441 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:07.177610 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:07.178102 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:07.178171 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.178251 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:07.178275 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:07.185384 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39855
I20260619 20:55:07.185464 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39855 every 8 connection(s)
I20260619 20:55:07.185798 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:07.188788 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.178667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43585 (local address 127.0.98.62:35449)
0619 20:55:07.178720 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.178723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.178939 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:55:07.179462 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.179463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.179478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.179543 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.180266 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.181093 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.182771 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.183130 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.183205 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.185672 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.185673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.185673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.185682 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.187281 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.187764 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.187765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.187766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.187826 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.188117 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.188118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.188118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.188235 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.188322 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.188634 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.188723 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:55:07.188901 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.178643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.1:43585)
0619 20:55:07.178818 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.178850 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.179334 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.179630 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.179634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.179996 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.180168 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.180173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.181198 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.181200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.182682 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.182686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.183259 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.183260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.183416 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.183864 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.183875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.185579 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.187399 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.187405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.187407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.187666 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.187930 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.187930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.187931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.188039 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.188306 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.188312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.188449 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.188567 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.188709 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260619 20:55:07.189368 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260619 20:55:07.189446 2326 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:07.189617 2326 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
I20260619 20:55:07.189787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2195
I20260619 20:55:07.189882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:07.190059 392 external_mini_cluster.cc:1428] Setting key d6ca5db860d612a72b4b29be59ed29bc
I20260619 20:55:07.190135 2123 ts_manager.cc:194] Registered new tserver with Master: b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855)
I20260619 20:55:07.191165 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43585
I20260619 20:55:07.196249 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.190477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44689 (local address 127.0.98.62:35449)
0619 20:55:07.190510 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.190512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.190580 (+ 68us) server_negotiation.cc:407] Connection header received
0619 20:55:07.190657 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.190658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.190667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.190713 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.191046 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.191601 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.192444 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.192750 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.192825 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.193690 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.193692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.193692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.193698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.195123 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.195488 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.195489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.195489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.195540 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.195751 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.195752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.195752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.195858 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.195932 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.196092 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.196162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:07.196266 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.190329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.1:44689)
0619 20:55:07.190530 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.190540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.190602 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.190751 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.190752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.190908 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.190954 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.190955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.191713 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.191714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.192357 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.192359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.192834 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.192834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.192886 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.193315 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.193319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.193578 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.195229 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.195230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.195230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.195401 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.195618 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.195619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.195619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.195676 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.195937 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.195938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.195978 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.196067 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.196163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":32,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:07.214764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:07.294395 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:07.294595 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:07.294620 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:07.295679 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:07.295712 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:07.295732 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:55:07.295744 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:07.295761 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:07.297113 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:07.297298 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:07.297465 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:07.302409 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.302520 2335 server_base.cc:1034] running on GCE node
W20260619 20:55:07.302416 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:07.302436 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.302822 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:07.303020 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:07.304193 2335 hybrid_clock.cc:648] HybridClock initialized: now 1781902507304182 us; error 31 us; skew 500 ppm
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:07.306571 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:07.307122 2335 webserver.cc:460] Webserver started at http://127.0.98.2:41711/ using document root <none> and password file <none>
I20260619 20:55:07.307363 2335 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:07.307407 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:07.307578 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:07.308622 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ec0a26d95ff4e32bc5006be8a154024"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "5cedc0ef21bc8ae0fb1ea7149f109d0b"
server_key_iv: "7e98a49c880cf3bd9e0ade6c055ae4e0"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.309015 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ec0a26d95ff4e32bc5006be8a154024"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "5cedc0ef21bc8ae0fb1ea7149f109d0b"
server_key_iv: "7e98a49c880cf3bd9e0ade6c055ae4e0"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.310341 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:55:07.311058 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.311235 2335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:07.311314 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "5ec0a26d95ff4e32bc5006be8a154024"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "5cedc0ef21bc8ae0fb1ea7149f109d0b"
server_key_iv: "7e98a49c880cf3bd9e0ade6c055ae4e0"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.311373 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:07.319025 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:07.321218 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:07.321323 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:07.321789 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:07.321834 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.321868 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:07.321882 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.328950 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46407
I20260619 20:55:07.329032 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46407 every 8 connection(s)
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:07.329345 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:07.332386 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.322262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56967 (local address 127.0.98.62:35449)
0619 20:55:07.322343 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.322346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.322619 (+ 273us) server_negotiation.cc:407] Connection header received
0619 20:55:07.323082 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.323084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.323098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.323180 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.324014 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.324882 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.326504 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.326858 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.326941 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.329684 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.329686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.329686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.329693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.331197 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.331565 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.331566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.331566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.331614 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.331823 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.331823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.331823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.331931 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.331980 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.332257 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.332315 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:55:07.332530 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.322273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.2:56967)
0619 20:55:07.322485 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.322523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.322985 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.323231 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.323233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.323616 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.323903 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.323909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.325025 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.325026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.326411 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.326414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.326957 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.326958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.327095 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.327600 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.327610 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.329578 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.331272 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.331277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.331286 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.331494 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.331673 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.331673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.331674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.331764 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.331989 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.331992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.332102 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.332201 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.332327 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":128,"threads_started":1}
I20260619 20:55:07.333019 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260619 20:55:07.333102 2466 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:07.333292 2466 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
I20260619 20:55:07.333653 2122 ts_manager.cc:194] Registered new tserver with Master: 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407)
I20260619 20:55:07.334040 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56967
I20260619 20:55:07.339416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2335
I20260619 20:55:07.339511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:07.339637 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.334068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.2:37385)
0619 20:55:07.334444 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.334458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.334535 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.334658 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.334658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.334867 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.334933 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.334935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.335533 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.335533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.336013 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.336014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.336377 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.336378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.336430 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.336783 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.336787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.337036 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.338693 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.338693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.338694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.338843 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.338986 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.338987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.338987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.339033 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.339257 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.339258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.339337 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.339423 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.339529 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":87,"threads_started":1}
I20260619 20:55:07.339634 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.334168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37385 (local address 127.0.98.62:35449)
0619 20:55:07.334388 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.334389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.334516 (+ 127us) server_negotiation.cc:407] Connection header received
0619 20:55:07.334598 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.334599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.334606 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.334649 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.335012 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.335442 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.336104 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.336304 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.336379 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.337130 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.337131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.337131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.337137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.338614 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.338893 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.338893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.338894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.338933 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.339077 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.339078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.339078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.339193 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.339239 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.339437 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.339529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260619 20:55:07.339712 392 external_mini_cluster.cc:1428] Setting key 76c7eac50b96a0cad1348d3eb53ab721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:07.365973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:07.447168 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:07.447341 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:07.447363 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:07.448426 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:07.448459 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:07.448479 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:55:07.448490 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:07.448508 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:07.449913 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:07.450088 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:07.450264 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:07.454895 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:07.454909 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.454931 2475 server_base.cc:1034] running on GCE node
W20260619 20:55:07.454885 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:07.455263 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:07.455513 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:07.456678 2475 hybrid_clock.cc:648] HybridClock initialized: now 1781902507456665 us; error 27 us; skew 500 ppm
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:07.459197 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:07.459564 2475 webserver.cc:460] Webserver started at http://127.0.98.3:33569/ using document root <none> and password file <none>
I20260619 20:55:07.459794 2475 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:07.459836 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:07.459969 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:07.460999 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "04f375e4b8144051876a866e027cc56e"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "63aea84ebd7e878338f2d46184e9134d"
server_key_iv: "e7a04b8f920e11eed5d504c6c91ddc72"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.461325 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04f375e4b8144051876a866e027cc56e"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "63aea84ebd7e878338f2d46184e9134d"
server_key_iv: "e7a04b8f920e11eed5d504c6c91ddc72"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.462641 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:07.463601 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.463816 2475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:07.463887 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "04f375e4b8144051876a866e027cc56e"
format_stamp: "Formatted at 2026-06-19 20:55:07 on dist-test-slave-8bgz"
server_key: "63aea84ebd7e878338f2d46184e9134d"
server_key_iv: "e7a04b8f920e11eed5d504c6c91ddc72"
server_key_version: "encryptionkey@0"
I20260619 20:55:07.463980 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:07.475319 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:07.477514 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:07.477667 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:07.478116 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:07.478153 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.478174 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:07.478211 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.485101 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260619 20:55:07.485150 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260619 20:55:07.485502 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:07.488888 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.478609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49773 (local address 127.0.98.62:35449)
0619 20:55:07.478673 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.478675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.478885 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:55:07.479322 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.479323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.479337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.479411 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.479951 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.480817 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.482562 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.482902 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.482971 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.486167 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.486168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.486169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.486176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.487669 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.488058 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.488059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.488059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.488110 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.488376 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.488377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.488377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.488501 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.488550 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.488761 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.488829 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:07.489053 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.478580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.3:49773)
0619 20:55:07.478744 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.478798 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.479233 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.479474 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.479477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.479685 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.479852 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.479855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.480915 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.480917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.482452 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.482455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.483011 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.483012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.483173 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.484171 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.484181 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.486056 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.487748 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.487754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.487755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.487990 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.488171 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.488171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.488171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.488311 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.488544 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.488548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.488653 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.488742 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.488853 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260619 20:55:07.489553 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260619 20:55:07.489658 2606 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:07.489838 2606 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
I20260619 20:55:07.490253 2122 ts_manager.cc:194] Registered new tserver with Master: 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:07.490401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2475
I20260619 20:55:07.490497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:07.490690 392 external_mini_cluster.cc:1428] Setting key 498482649754ada912d8fe4baec33967
I20260619 20:55:07.490772 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49773
I20260619 20:55:07.491654 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:55:07.496577 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.490615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34333 (local address 127.0.98.62:35449)
0619 20:55:07.490649 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.490650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.490777 (+ 127us) server_negotiation.cc:407] Connection header received
0619 20:55:07.490884 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.490885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.490898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.490963 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.491402 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.491935 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.492621 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.492829 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.492915 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.493834 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.493835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.493836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.493842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.495493 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.495804 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.495805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.495806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.495865 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.496045 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.496046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.496047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.496175 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.496248 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.496394 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.496481 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:07.496583 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.490543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.3:34333)
0619 20:55:07.490724 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.490734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.490799 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.491020 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.491020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.491243 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.491309 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.491311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.492026 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.492026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.492533 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.492535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.492916 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.492917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.492987 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.493447 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.493453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.493720 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.495571 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.495572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.495572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.495726 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.495932 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.495933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.495933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.495979 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.496261 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.496261 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.496293 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.496343 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.496467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:07.514302 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.507632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.98.62:35449)
0619 20:55:07.507692 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.507694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.507715 (+ 21us) server_negotiation.cc:407] Connection header received
0619 20:55:07.507777 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.507778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.507785 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.507833 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.508369 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.508805 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.509470 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.509637 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.509684 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.511767 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.511768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.511768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.511774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.513104 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.513404 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.513405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.513405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.513452 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.513733 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.513734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.513734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.513823 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.513938 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.514096 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.514153 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:07.515800 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:07.517277 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:07.523339 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.0.98.3:37195)
0619 20:55:07.519660 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.519664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.519735 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:55:07.519804 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.519806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.519830 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.519921 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:07.520079 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.520821 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.522241 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.522959 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523036 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523093 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.523222 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.523256 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260619 20:55:07.523329 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.0.1:50004)
0619 20:55:07.519668 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.519677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.519746 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.519930 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.519931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.519933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:07.519979 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.519980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.520925 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.520926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.522131 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.522134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.523047 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.523120 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523132 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.523194 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.523254 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":39,"threads_started":1}
I20260619 20:55:07.523607 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46407 (local address 127.0.0.1:44164)
0619 20:55:07.519464 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.519474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.519522 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.519805 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.519808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.519810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:07.519867 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.519869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.521247 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.521248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.522345 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.522347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.523392 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.523462 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523466 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.523510 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.523537 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":24,"thread_start_us":52,"threads_started":1}
I20260619 20:55:07.523607 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.0.1:34986)
0619 20:55:07.519339 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.519372 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.519424 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.520139 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.520140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.520142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:07.520192 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.520193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.521431 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.521432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.522554 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.522556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.523402 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.523454 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523457 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.523486 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.523527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":50,"threads_started":1}
I20260619 20:55:07.524073 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.0.98.1:39855)
0619 20:55:07.519901 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.519906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.519914 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:07.520012 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.520014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.520042 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.520264 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:07.520328 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.521338 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.522624 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.523315 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523854 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523922 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.523947 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.523984 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":70,"threads_started":1}
I20260619 20:55:07.524268 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 (DEFAULT_TABLE table=test-table [id=260b6edf1ed049d983bca68c9757bc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:07.524686 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b040d7d2f94e53bcd9fa48a4c61378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.524784 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 (DEFAULT_TABLE table=test-table [id=260b6edf1ed049d983bca68c9757bc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:07.525079 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b040d7d2f94e53bcd9fa48a4c61378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.526372 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.519290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.0.98.2:46407)
0619 20:55:07.519621 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.519625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.519632 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:07.519660 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.519661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.519672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.519779 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:07.519949 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.521158 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.522518 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.523223 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.523291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.523358 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.523736 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.523786 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":89,"threads_started":1}
I20260619 20:55:07.526934 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 (DEFAULT_TABLE table=test-table [id=260b6edf1ed049d983bca68c9757bc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:07.527245 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b040d7d2f94e53bcd9fa48a4c61378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.528465 2627 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Bootstrap starting.
I20260619 20:55:07.529068 2627 tablet_bootstrap.cc:654] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.529451 2627 log.cc:824] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:07.530130 2627 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: No bootstrap required, opened a new log
I20260619 20:55:07.530210 2627 ts_tablet_manager.cc:1397] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:07.530583 2629 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Bootstrap starting.
I20260619 20:55:07.530686 2626 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Bootstrap starting.
I20260619 20:55:07.531221 2629 tablet_bootstrap.cc:654] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.531481 2626 tablet_bootstrap.cc:654] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.531544 2629 log.cc:824] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:07.531812 2626 log.cc:824] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:07.531910 2627 raft_consensus.cc:348] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.532080 2627 raft_consensus.cc:374] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.532148 2627 raft_consensus.cc:729] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.532263 2627 consensus_queue.cc:260] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.532326 2629 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: No bootstrap required, opened a new log
I20260619 20:55:07.532404 2629 ts_tablet_manager.cc:1397] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:07.532533 2627 ts_tablet_manager.cc:1428] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:07.532582 2326 heartbeater.cc:503] Master 127.0.98.62:35449 was elected leader, sending a full tablet report...
I20260619 20:55:07.534307 2629 raft_consensus.cc:348] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.534446 2629 raft_consensus.cc:374] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.534493 2629 raft_consensus.cc:729] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.534606 2629 consensus_queue.cc:260] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.534760 2626 tablet_bootstrap.cc:492] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: No bootstrap required, opened a new log
I20260619 20:55:07.534845 2626 ts_tablet_manager.cc:1397] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:55:07.534878 2629 ts_tablet_manager.cc:1428] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:55:07.534881 2466 heartbeater.cc:503] Master 127.0.98.62:35449 was elected leader, sending a full tablet report...
I20260619 20:55:07.536474 2626 raft_consensus.cc:348] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.536616 2626 raft_consensus.cc:374] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.536660 2626 raft_consensus.cc:729] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f375e4b8144051876a866e027cc56e, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.536772 2626 consensus_queue.cc:260] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.537014 2626 ts_tablet_manager.cc:1428] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:07.537074 2606 heartbeater.cc:503] Master 127.0.98.62:35449 was elected leader, sending a full tablet report...
W20260619 20:55:07.579886 2467 tablet.cc:2369] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:07.687014 2327 tablet.cc:2369] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:07.736054 2607 tablet.cc:2369] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:07.866425 2633 raft_consensus.cc:482] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:07.866621 2633 raft_consensus.cc:504] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.866941 2633 leader_election.cc:288] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:07.869910 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.867197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44869 (local address 127.0.98.1:39855)
0619 20:55:07.867257 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.867259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.867374 (+ 115us) server_negotiation.cc:407] Connection header received
0619 20:55:07.867437 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.867438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.867449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.867523 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:07.867714 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.868203 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869150 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.869508 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.869635 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.869790 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.869829 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:55:07.869925 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.867477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45199 (local address 127.0.98.3:37195)
0619 20:55:07.867545 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.867547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.867558 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:07.867584 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.867584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.867594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.867643 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:07.867801 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.868284 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869162 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.869540 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869654 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.869663 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.869820 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.869856 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:55:07.869941 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.867140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.98.2:44869)
0619 20:55:07.867291 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.867302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.867368 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.867571 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.867572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.867574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:07.867639 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.867640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.868291 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.868292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.869052 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.869054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.869630 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.869708 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.869712 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.869786 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.869824 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
I20260619 20:55:07.869949 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.867362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.2:45199)
0619 20:55:07.867473 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.867481 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.867522 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.867661 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.867661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.867664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:07.867733 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.867734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.868379 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.868380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.869075 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.869076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.869657 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.869658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.869728 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.869732 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.869812 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.869853 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":39,"threads_started":1}
I20260619 20:55:07.870316 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b040d7d2f94e53bcd9fa48a4c61378" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e" is_pre_election: true
I20260619 20:55:07.870487 2560 raft_consensus.cc:2454] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:07.870472 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b040d7d2f94e53bcd9fa48a4c61378" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a" is_pre_election: true
I20260619 20:55:07.870596 2280 raft_consensus.cc:2454] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:07.870697 2351 leader_election.cc:302] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f375e4b8144051876a866e027cc56e, 5ec0a26d95ff4e32bc5006be8a154024; no voters:
I20260619 20:55:07.870800 2633 raft_consensus.cc:2781] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:07.870851 2633 raft_consensus.cc:482] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:07.870877 2633 raft_consensus.cc:3037] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:07.871503 2633 raft_consensus.cc:504] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.871629 2633 leader_election.cc:288] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Requested vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:07.871803 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b040d7d2f94e53bcd9fa48a4c61378" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e"
I20260619 20:55:07.871817 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b040d7d2f94e53bcd9fa48a4c61378" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a"
I20260619 20:55:07.871920 2560 raft_consensus.cc:3037] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:07.871918 2280 raft_consensus.cc:3037] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:07.872610 2280 raft_consensus.cc:2454] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:07.872761 2560 raft_consensus.cc:2454] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:07.872772 2351 leader_election.cc:302] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:07.872848 2633 raft_consensus.cc:2781] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:07.872934 2633 raft_consensus.cc:686] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Running, Role: LEADER
I20260619 20:55:07.873028 2633 consensus_queue.cc:237] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.873683 2122 catalog_manager.cc:5495] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:55:07 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:07.981271 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.974945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.0.1:40722)
0619 20:55:07.975079 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:07.975095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:07.975152 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:07.975272 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:07.975272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:07.975557 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:07.975600 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.975601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.976183 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.976184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.976699 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:07.976700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.977000 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.977000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:07.977037 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.977516 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:07.977520 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:07.978994 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:07.980440 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.980445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.980447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.980626 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.980793 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:07.980793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:07.980793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:07.980845 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:07.981039 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:07.981043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:07.981080 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:55:07.981130 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:55:07.981209 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260619 20:55:07.981331 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:07.975037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.0.98.62:35449)
0619 20:55:07.975102 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:07.975103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:07.975152 (+ 49us) server_negotiation.cc:407] Connection header received
0619 20:55:07.975191 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:07.975192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:07.975200 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:07.975249 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:07.975660 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.976121 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.976769 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:07.976937 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:07.976984 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:07.979079 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:07.979081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:07.979081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:07.979088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:07.980360 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.980688 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.980689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.980689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.980736 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:07.980896 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:07.980897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:07.980897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:07.980987 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:07.981033 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:07.981161 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 20:55:07.981253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":45}
I20260619 20:55:07.983317 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40722:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:55:07.983513 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:07.984905 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 5c116e638f884cf8ace3b616994fee0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cd4cbae5c654232a071abeaddb5a5b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:07.984928 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 5c116e638f884cf8ace3b616994fee0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cd4cbae5c654232a071abeaddb5a5b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:07.984935 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 5c116e638f884cf8ace3b616994fee0e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cd4cbae5c654232a071abeaddb5a5b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:07.985116 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c116e638f884cf8ace3b616994fee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.985111 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c116e638f884cf8ace3b616994fee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.985129 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c116e638f884cf8ace3b616994fee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:07.986276 2626 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: Bootstrap starting.
I20260619 20:55:07.986596 2629 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: Bootstrap starting.
I20260619 20:55:07.986608 2627 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: Bootstrap starting.
I20260619 20:55:07.986836 2626 tablet_bootstrap.cc:654] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.986990 2627 tablet_bootstrap.cc:654] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.986999 2629 tablet_bootstrap.cc:654] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:07.987560 2626 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: No bootstrap required, opened a new log
I20260619 20:55:07.987612 2627 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: No bootstrap required, opened a new log
I20260619 20:55:07.987612 2629 tablet_bootstrap.cc:492] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: No bootstrap required, opened a new log
I20260619 20:55:07.987633 2626 ts_tablet_manager.cc:1397] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:07.987653 2629 ts_tablet_manager.cc:1397] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:07.987670 2627 ts_tablet_manager.cc:1397] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:07.987782 2626 raft_consensus.cc:348] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.987834 2626 raft_consensus.cc:374] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.987847 2626 raft_consensus.cc:729] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f375e4b8144051876a866e027cc56e, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.987840 2627 raft_consensus.cc:348] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.987911 2627 raft_consensus.cc:374] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.987905 2626 consensus_queue.cc:260] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.987934 2627 raft_consensus.cc:729] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.987988 2626 ts_tablet_manager.cc:1428] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.987974 2627 consensus_queue.cc:260] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.987994 2629 raft_consensus.cc:348] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.988080 2629 raft_consensus.cc:374] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:07.988102 2629 raft_consensus.cc:729] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Initialized, Role: FOLLOWER
I20260619 20:55:07.988102 2627 ts_tablet_manager.cc:1428] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:07.988145 2629 consensus_queue.cc:260] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:07.988229 2629 ts_tablet_manager.cc:1428] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:07.988245 2634 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:07.988329 2634 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:07.988317 2633 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:07.988372 2633 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:07.988482 2632 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:07.988564 2632 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:08.343078 2633 consensus_queue.cc:1035] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.345489 2650 consensus_queue.cc:1035] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.363891 2650 raft_consensus.cc:482] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:08.363986 2650 raft_consensus.cc:504] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.364161 2650 leader_election.cc:288] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:08.364398 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c116e638f884cf8ace3b616994fee0e" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e" is_pre_election: true
I20260619 20:55:08.364377 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c116e638f884cf8ace3b616994fee0e" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a" is_pre_election: true
I20260619 20:55:08.364475 2560 raft_consensus.cc:2454] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:08.364497 2280 raft_consensus.cc:2454] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:08.364627 2351 leader_election.cc:302] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f375e4b8144051876a866e027cc56e, 5ec0a26d95ff4e32bc5006be8a154024; no voters:
I20260619 20:55:08.364749 2650 raft_consensus.cc:2781] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:08.364813 2650 raft_consensus.cc:482] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:08.364845 2650 raft_consensus.cc:3037] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.434266 2650 raft_consensus.cc:504] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.434477 2650 leader_election.cc:288] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Requested vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:08.434720 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c116e638f884cf8ace3b616994fee0e" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a"
I20260619 20:55:08.434832 2280 raft_consensus.cc:3037] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.434778 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c116e638f884cf8ace3b616994fee0e" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e"
I20260619 20:55:08.434887 2560 raft_consensus.cc:3037] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.435482 2280 raft_consensus.cc:2454] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:08.435690 2351 leader_election.cc:302] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:08.435786 2560 raft_consensus.cc:2454] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:08.435798 2650 raft_consensus.cc:2781] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:08.435853 2650 raft_consensus.cc:686] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Running, Role: LEADER
I20260619 20:55:08.435899 2650 consensus_queue.cc:237] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.436200 2637 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: TxnStatusTablet state changed. Reason: New leader 5ec0a26d95ff4e32bc5006be8a154024. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.436282 2637 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:08.436483 2655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:55:08.436492 2122 catalog_manager.cc:5495] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:08.436635 2655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:55:08.538911 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:08.535743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.98.2:46407)
0619 20:55:08.535875 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:08.535877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:08.535886 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:08.535931 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:08.535932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:08.535950 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:08.536051 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:08.536210 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.537258 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.538019 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.538249 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.538329 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:08.538527 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:08.538621 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:08.538724 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:08.538774 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:55:08.538794 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260619 20:55:08.539054 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:08.535641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46407 (local address 127.0.0.1:44170)
0619 20:55:08.535760 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:08.535776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:08.535851 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:08.536062 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:08.536064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:08.536065 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:08.536124 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:08.536126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.537327 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.537329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:08.537917 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:08.537919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.538341 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.538342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:08.538406 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:08.538418 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:08.538705 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:08.538715 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 20:55:08.538755 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 20:55:08.538972 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260619 20:55:08.540688 2560 raft_consensus.cc:1261] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:08.540704 2280 raft_consensus.cc:1261] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:08.540937 2650 consensus_queue.cc:1035] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.541049 2637 consensus_queue.cc:1035] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.542013 2634 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: TxnStatusTablet state changed. Reason: New leader 5ec0a26d95ff4e32bc5006be8a154024. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.542114 2634 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:08.542093 2632 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: TxnStatusTablet state changed. Reason: New leader 5ec0a26d95ff4e32bc5006be8a154024. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.542183 2632 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:08.542999 2637 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.543076 2637 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:08.543174 2651 mvcc.cc:205] Tried to move back new op lower bound from 7298672674980315136 to 7298672674553827328. Current Snapshot: MvccSnapshot[applied={T|T < 7298672674980315136}]
I20260619 20:55:08.543294 2634 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.543346 2634 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 04f375e4b8144051876a866e027cc56e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:08.544045 2637 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.544128 2637 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P 5ec0a26d95ff4e32bc5006be8a154024: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:08.544190 2653 mvcc.cc:205] Tried to move back new op lower bound from 7298672674980315136 to 7298672674553827328. Current Snapshot: MvccSnapshot[applied={T|T < 7298672674980315136}]
I20260619 20:55:08.544267 2632 tablet_replica.cc:439] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } }
I20260619 20:55:08.544358 2632 tablet_replica.cc:442] T 5c116e638f884cf8ace3b616994fee0e P b6621e97316e4b4084545187b8f1310a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:08.547930 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:08.545358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.98.2:46407)
0619 20:55:08.545432 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:08.545434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:08.545532 (+ 98us) server_negotiation.cc:407] Connection header received
0619 20:55:08.545585 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:08.545586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:08.545595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:08.545650 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:08.545756 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.546478 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.547232 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.547524 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.547593 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:08.547723 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:08.547753 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:08.547812 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:08.547846 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 20:55:08.547863 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:55:08.552812 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:08.550275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46407 (local address 127.0.98.2:60745)
0619 20:55:08.550399 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:08.550408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:08.550480 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:08.550605 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:08.550605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:08.550606 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:08.550636 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:08.550637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.551412 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.551413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:08.551949 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:08.551950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.552439 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.552440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:08.552475 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:08.552486 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:08.552679 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:08.552681 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:08.552715 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:55:08.552745 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260619 20:55:08.552819 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:08.550278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60745 (local address 127.0.98.2:46407)
0619 20:55:08.550313 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:08.550315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:08.550517 (+ 202us) server_negotiation.cc:407] Connection header received
0619 20:55:08.550538 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:08.550539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:08.550547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:08.550602 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:08.550673 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.551341 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.552027 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:08.552351 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:08.552437 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:08.552579 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:08.552613 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:08.552670 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:08.552717 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 20:55:08.552736 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260619 20:55:08.564985 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:08.565095 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:08.565810 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Sending DeleteTablet for 3 replicas of tablet 00b040d7d2f94e53bcd9fa48a4c61378
I20260619 20:55:08.566126 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50004
I20260619 20:55:08.566140 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260619 20:55:08.566334 2671 tablet_replica.cc:330] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: stopping tablet replica
I20260619 20:55:08.566401 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b040d7d2f94e53bcd9fa48a4c61378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44164
I20260619 20:55:08.566495 2671 raft_consensus.cc:2229] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:08.566598 2671 raft_consensus.cc:2258] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:08.566632 2673 tablet_replica.cc:330] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: stopping tablet replica
I20260619 20:55:08.566793 2673 raft_consensus.cc:2229] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:08.566816 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:08.566946 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:08.566947 2673 raft_consensus.cc:2258] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:08.567086 2671 ts_tablet_manager.cc:1905] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:08.567416 2672 tablet_replica.cc:330] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: stopping tablet replica
I20260619 20:55:08.567557 2672 raft_consensus.cc:2229] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:08.567656 2672 raft_consensus.cc:2258] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:08.568020 2672 ts_tablet_manager.cc:1905] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:08.568495 2673 ts_tablet_manager.cc:1905] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:08.568521 2540 tablet_service.cc:1467] Processing CreateTablet for tablet e00d371edb8644949b0f32fa84d26380 (DEFAULT_TABLE table=test-table [id=5491a15cff42440094ee24aefe098d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:08.568585 2260 tablet_service.cc:1467] Processing CreateTablet for tablet e00d371edb8644949b0f32fa84d26380 (DEFAULT_TABLE table=test-table [id=5491a15cff42440094ee24aefe098d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:08.568624 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00d371edb8644949b0f32fa84d26380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:08.568697 2399 tablet_service.cc:1467] Processing CreateTablet for tablet e00d371edb8644949b0f32fa84d26380 (DEFAULT_TABLE table=test-table [id=5491a15cff42440094ee24aefe098d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:08.568786 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00d371edb8644949b0f32fa84d26380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:08.570111 2674 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Bootstrap starting.
I20260619 20:55:08.570490 2674 tablet_bootstrap.cc:654] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:08.571040 2674 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: No bootstrap required, opened a new log
I20260619 20:55:08.571103 2674 ts_tablet_manager.cc:1397] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:08.571250 2674 raft_consensus.cc:348] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.571323 2674 raft_consensus.cc:374] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:08.571355 2674 raft_consensus.cc:729] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f375e4b8144051876a866e027cc56e, State: Initialized, Role: FOLLOWER
I20260619 20:55:08.571405 2674 consensus_queue.cc:260] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.571496 2674 ts_tablet_manager.cc:1428] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:08.600739 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00d371edb8644949b0f32fa84d26380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:08.602653 2676 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Bootstrap starting.
I20260619 20:55:08.603163 2676 tablet_bootstrap.cc:654] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:08.603283 2671 ts_tablet_manager.cc:1918] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:08.603423 2671 log.cc:1196] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/00b040d7d2f94e53bcd9fa48a4c61378
I20260619 20:55:08.603811 2676 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: No bootstrap required, opened a new log
I20260619 20:55:08.603870 2676 ts_tablet_manager.cc:1397] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:08.604003 2676 raft_consensus.cc:348] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.604066 2676 raft_consensus.cc:374] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:08.604097 2676 raft_consensus.cc:729] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Initialized, Role: FOLLOWER
I20260619 20:55:08.604158 2676 consensus_queue.cc:260] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.604241 2676 ts_tablet_manager.cc:1428] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:08.603803 2671 ts_tablet_manager.cc:1939] T 00b040d7d2f94e53bcd9fa48a4c61378 P b6621e97316e4b4084545187b8f1310a: Deleting consensus metadata
I20260619 20:55:08.609870 2673 ts_tablet_manager.cc:1918] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:08.609961 2673 log.cc:1196] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/00b040d7d2f94e53bcd9fa48a4c61378
I20260619 20:55:08.610235 2673 ts_tablet_manager.cc:1939] T 00b040d7d2f94e53bcd9fa48a4c61378 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting consensus metadata
I20260619 20:55:08.610894 2678 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Bootstrap starting.
I20260619 20:55:08.611232 2678 tablet_bootstrap.cc:654] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:08.611801 2678 tablet_bootstrap.cc:492] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: No bootstrap required, opened a new log
I20260619 20:55:08.611865 2678 ts_tablet_manager.cc:1397] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:08.611972 2678 raft_consensus.cc:348] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.612089 2678 raft_consensus.cc:374] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:08.612113 2678 raft_consensus.cc:729] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Initialized, Role: FOLLOWER
I20260619 20:55:08.612211 2678 consensus_queue.cc:260] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.612337 2678 ts_tablet_manager.cc:1428] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:08.612418 2110 catalog_manager.cc:4841] TS 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407): tablet 00b040d7d2f94e53bcd9fa48a4c61378 (table test-table [id=260b6edf1ed049d983bca68c9757bc4b]) successfully deleted
I20260619 20:55:08.642629 2672 ts_tablet_manager.cc:1918] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:08.642745 2672 log.cc:1196] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/00b040d7d2f94e53bcd9fa48a4c61378
I20260619 20:55:08.643029 2672 ts_tablet_manager.cc:1939] T 00b040d7d2f94e53bcd9fa48a4c61378 P 04f375e4b8144051876a866e027cc56e: Deleting consensus metadata
I20260619 20:55:08.642983 2107 catalog_manager.cc:4841] TS b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855): tablet 00b040d7d2f94e53bcd9fa48a4c61378 (table test-table [id=260b6edf1ed049d983bca68c9757bc4b]) successfully deleted
I20260619 20:55:08.645720 2107 catalog_manager.cc:4841] TS 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195): tablet 00b040d7d2f94e53bcd9fa48a4c61378 (table test-table [id=260b6edf1ed049d983bca68c9757bc4b]) successfully deleted
I20260619 20:55:08.659565 2633 raft_consensus.cc:482] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:08.659718 2633 raft_consensus.cc:504] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.660408 2633 leader_election.cc:288] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:08.660568 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00d371edb8644949b0f32fa84d26380" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a" is_pre_election: true
I20260619 20:55:08.660671 2280 raft_consensus.cc:2454] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:08.660600 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00d371edb8644949b0f32fa84d26380" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e" is_pre_election: true
I20260619 20:55:08.660732 2560 raft_consensus.cc:2454] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:08.660818 2351 leader_election.cc:302] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:08.660908 2633 raft_consensus.cc:2781] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:08.660964 2633 raft_consensus.cc:482] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:08.660992 2633 raft_consensus.cc:3037] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.697769 2633 raft_consensus.cc:504] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.697989 2633 leader_election.cc:288] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Requested vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:08.698277 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00d371edb8644949b0f32fa84d26380" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a"
I20260619 20:55:08.698329 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00d371edb8644949b0f32fa84d26380" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e"
I20260619 20:55:08.698412 2280 raft_consensus.cc:3037] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.698421 2560 raft_consensus.cc:3037] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:08.699049 2280 raft_consensus.cc:2454] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:08.699290 2351 leader_election.cc:302] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:08.699347 2560 raft_consensus.cc:2454] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:08.699409 2633 raft_consensus.cc:2781] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:08.699452 2633 raft_consensus.cc:686] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Running, Role: LEADER
I20260619 20:55:08.699502 2633 consensus_queue.cc:237] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:08.700138 2122 catalog_manager.cc:5495] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:08.721014 2560 raft_consensus.cc:1261] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:08.721021 2280 raft_consensus.cc:1261] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:08.721423 2633 consensus_queue.cc:1035] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.721525 2637 consensus_queue.cc:1035] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:08.724615 2649 mvcc.cc:205] Tried to move back new op lower bound from 7298672675719483392 to 7298672675633205248. Current Snapshot: MvccSnapshot[applied={T|T < 7298672675719483392}]
W20260619 20:55:08.728886 2649 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:55:10.249284 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:10.249410 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:10.250241 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Sending DeleteTablet for 3 replicas of tablet e00d371edb8644949b0f32fa84d26380
I20260619 20:55:10.250530 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet e00d371edb8644949b0f32fa84d26380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44164
I20260619 20:55:10.250567 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet e00d371edb8644949b0f32fa84d26380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260619 20:55:10.250582 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet e00d371edb8644949b0f32fa84d26380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50004
I20260619 20:55:10.250774 2704 tablet_replica.cc:330] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: stopping tablet replica
I20260619 20:55:10.250833 2704 raft_consensus.cc:2229] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:10.250869 2704 raft_consensus.cc:2258] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.251121 2704 ts_tablet_manager.cc:1905] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.253062 2704 ts_tablet_manager.cc:1918] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:55:10.253155 2704 log.cc:1196] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/e00d371edb8644949b0f32fa84d26380
I20260619 20:55:10.253263 2703 tablet_replica.cc:330] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: stopping tablet replica
I20260619 20:55:10.253340 2703 raft_consensus.cc:2229] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:10.253399 2703 raft_consensus.cc:2258] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.253474 2704 ts_tablet_manager.cc:1939] T e00d371edb8644949b0f32fa84d26380 P 04f375e4b8144051876a866e027cc56e: Deleting consensus metadata
I20260619 20:55:10.253507 2705 tablet_replica.cc:330] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: stopping tablet replica
I20260619 20:55:10.253654 2705 raft_consensus.cc:2229] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:10.253648 2703 ts_tablet_manager.cc:1905] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.253746 2705 raft_consensus.cc:2258] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.253856 2107 catalog_manager.cc:4841] TS 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195): tablet e00d371edb8644949b0f32fa84d26380 (table test-table [id=5491a15cff42440094ee24aefe098d52]) successfully deleted
I20260619 20:55:10.254057 2705 ts_tablet_manager.cc:1905] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.255215 2703 ts_tablet_manager.cc:1918] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:55:10.255308 2703 log.cc:1196] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/e00d371edb8644949b0f32fa84d26380
I20260619 20:55:10.255676 2703 ts_tablet_manager.cc:1939] T e00d371edb8644949b0f32fa84d26380 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting consensus metadata
I20260619 20:55:10.255704 2705 ts_tablet_manager.cc:1918] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:55:10.255754 2705 log.cc:1196] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/e00d371edb8644949b0f32fa84d26380
I20260619 20:55:10.256022 2705 ts_tablet_manager.cc:1939] T e00d371edb8644949b0f32fa84d26380 P b6621e97316e4b4084545187b8f1310a: Deleting consensus metadata
I20260619 20:55:10.256032 2110 catalog_manager.cc:4841] TS 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407): tablet e00d371edb8644949b0f32fa84d26380 (table test-table [id=5491a15cff42440094ee24aefe098d52]) successfully deleted
I20260619 20:55:10.256316 2107 catalog_manager.cc:4841] TS b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855): tablet e00d371edb8644949b0f32fa84d26380 (table test-table [id=5491a15cff42440094ee24aefe098d52]) successfully deleted
Jun 19 20:55:10 dist-test-slave-8bgz krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902510, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:55:10 dist-test-slave-8bgz krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902510, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:10.274345 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.267234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.0.98.62:35449)
0619 20:55:10.267380 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.267382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.267456 (+ 74us) server_negotiation.cc:407] Connection header received
0619 20:55:10.267542 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.267542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.267554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.267639 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:10.268181 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.268709 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.269457 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.269654 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.269738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.271895 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:10.271896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:10.271896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:10.271904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:10.273143 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:10.273501 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:10.273502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:10.273503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:10.273573 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:10.273788 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:10.273789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:10.273789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:10.273900 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:10.274006 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.274106 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.274185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260619 20:55:10.275671 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:10.275843 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:10.277206 2399 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 (DEFAULT_TABLE table=test-table [id=014883fce9db4ecd97e525de1d77a8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.277264 2260 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 (DEFAULT_TABLE table=test-table [id=014883fce9db4ecd97e525de1d77a8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.277264 2540 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 (DEFAULT_TABLE table=test-table [id=014883fce9db4ecd97e525de1d77a8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.277369 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90fa9b35a4b8e81c63d74783144a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.277388 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90fa9b35a4b8e81c63d74783144a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.277400 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90fa9b35a4b8e81c63d74783144a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.278693 2715 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Bootstrap starting.
I20260619 20:55:10.278769 2714 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Bootstrap starting.
I20260619 20:55:10.279096 2714 tablet_bootstrap.cc:654] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.279112 2715 tablet_bootstrap.cc:654] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.279481 2716 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Bootstrap starting.
I20260619 20:55:10.279675 2714 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: No bootstrap required, opened a new log
I20260619 20:55:10.279716 2714 ts_tablet_manager.cc:1397] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:10.279814 2716 tablet_bootstrap.cc:654] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.279846 2714 raft_consensus.cc:348] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.279942 2714 raft_consensus.cc:374] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.279971 2714 raft_consensus.cc:729] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.280040 2714 consensus_queue.cc:260] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.280253 2714 ts_tablet_manager.cc:1428] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:10.280526 2715 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: No bootstrap required, opened a new log
I20260619 20:55:10.280566 2716 tablet_bootstrap.cc:492] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: No bootstrap required, opened a new log
I20260619 20:55:10.280602 2715 ts_tablet_manager.cc:1397] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:10.280611 2716 ts_tablet_manager.cc:1397] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:10.280812 2715 raft_consensus.cc:348] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.280922 2715 raft_consensus.cc:374] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.280895 2716 raft_consensus.cc:348] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.280972 2715 raft_consensus.cc:729] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.280982 2716 raft_consensus.cc:374] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.281006 2716 raft_consensus.cc:729] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f375e4b8144051876a866e027cc56e, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.281028 2715 consensus_queue.cc:260] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.281080 2716 consensus_queue.cc:260] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.281196 2715 ts_tablet_manager.cc:1428] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:55:10.281322 2716 ts_tablet_manager.cc:1428] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:10.341485 2698 raft_consensus.cc:482] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:10.341583 2698 raft_consensus.cc:504] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.341698 2698 leader_election.cc:288] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:10.341933 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90fa9b35a4b8e81c63d74783144a0" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a" is_pre_election: true
I20260619 20:55:10.341959 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90fa9b35a4b8e81c63d74783144a0" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e" is_pre_election: true
I20260619 20:55:10.342044 2280 raft_consensus.cc:2454] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:10.342048 2560 raft_consensus.cc:2454] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 0.
I20260619 20:55:10.342262 2351 leader_election.cc:302] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f375e4b8144051876a866e027cc56e, 5ec0a26d95ff4e32bc5006be8a154024; no voters:
I20260619 20:55:10.342397 2698 raft_consensus.cc:2781] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:10.342453 2698 raft_consensus.cc:482] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:10.342470 2698 raft_consensus.cc:3037] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.343183 2698 raft_consensus.cc:504] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.343281 2698 leader_election.cc:288] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Requested vote from peers b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:10.343441 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90fa9b35a4b8e81c63d74783144a0" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6621e97316e4b4084545187b8f1310a"
I20260619 20:55:10.343534 2280 raft_consensus.cc:3037] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.343501 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90fa9b35a4b8e81c63d74783144a0" candidate_uuid: "5ec0a26d95ff4e32bc5006be8a154024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e"
I20260619 20:55:10.343617 2560 raft_consensus.cc:3037] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.344172 2280 raft_consensus.cc:2454] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:10.344199 2560 raft_consensus.cc:2454] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec0a26d95ff4e32bc5006be8a154024 in term 1.
I20260619 20:55:10.344354 2351 leader_election.cc:302] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:10.344470 2698 raft_consensus.cc:2781] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:10.344533 2698 raft_consensus.cc:686] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Running, Role: LEADER
I20260619 20:55:10.344573 2698 consensus_queue.cc:237] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.345211 2123 catalog_manager.cc:5495] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5ec0a26d95ff4e32bc5006be8a154024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:10.355206 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.353001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.0.98.2:46407)
0619 20:55:10.353121 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.353122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.353129 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:10.353146 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.353147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.353156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.353208 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:10.353306 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.353777 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.354571 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.354757 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.354810 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.354971 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.354995 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.355033 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.355102 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.355119 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260619 20:55:10.357251 2560 raft_consensus.cc:1261] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:10.357290 2280 raft_consensus.cc:1261] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Refusing update from remote peer 5ec0a26d95ff4e32bc5006be8a154024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:10.357537 2686 consensus_queue.cc:1035] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:10.357638 2633 consensus_queue.cc:1035] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:10.366904 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:10.366981 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:10.367565 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Sending DeleteTablet for 3 replicas of tablet a9f90fa9b35a4b8e81c63d74783144a0
I20260619 20:55:10.367799 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260619 20:55:10.367821 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44164
I20260619 20:55:10.367899 2703 tablet_replica.cc:330] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: stopping tablet replica
I20260619 20:55:10.367889 2705 tablet_replica.cc:330] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: stopping tablet replica
I20260619 20:55:10.367913 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f90fa9b35a4b8e81c63d74783144a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50004
I20260619 20:55:10.367947 2703 raft_consensus.cc:2229] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:10.367952 2705 raft_consensus.cc:2229] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:10.368000 2705 raft_consensus.cc:2258] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.367995 2703 raft_consensus.cc:2258] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.368000 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:55:10.368072 2704 tablet_replica.cc:330] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: stopping tablet replica
W20260619 20:55:10.368119 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:10.368119 2704 raft_consensus.cc:2229] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:10.368178 2704 raft_consensus.cc:2258] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:10.368269 2705 ts_tablet_manager.cc:1905] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.368642 2703 ts_tablet_manager.cc:1905] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.368651 2704 ts_tablet_manager.cc:1905] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:10.370354 2704 ts_tablet_manager.cc:1918] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:10.370373 2705 ts_tablet_manager.cc:1918] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:10.370379 2703 ts_tablet_manager.cc:1918] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:10.370419 2705 log.cc:1196] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/a9f90fa9b35a4b8e81c63d74783144a0
I20260619 20:55:10.370419 2704 log.cc:1196] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/a9f90fa9b35a4b8e81c63d74783144a0
I20260619 20:55:10.370432 2703 log.cc:1196] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/a9f90fa9b35a4b8e81c63d74783144a0
I20260619 20:55:10.370637 2705 ts_tablet_manager.cc:1939] T a9f90fa9b35a4b8e81c63d74783144a0 P b6621e97316e4b4084545187b8f1310a: Deleting consensus metadata
I20260619 20:55:10.370637 2704 ts_tablet_manager.cc:1939] T a9f90fa9b35a4b8e81c63d74783144a0 P 04f375e4b8144051876a866e027cc56e: Deleting consensus metadata
I20260619 20:55:10.370657 2703 ts_tablet_manager.cc:1939] T a9f90fa9b35a4b8e81c63d74783144a0 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting consensus metadata
I20260619 20:55:10.370929 2260 tablet_service.cc:1467] Processing CreateTablet for tablet d65c58c88f674c8ea16946629d0ea461 (DEFAULT_TABLE table=test-table [id=949713c262424416bce3377c2d4be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.370971 2110 catalog_manager.cc:4841] TS 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407): tablet a9f90fa9b35a4b8e81c63d74783144a0 (table test-table [id=014883fce9db4ecd97e525de1d77a8cc]) successfully deleted
I20260619 20:55:10.371035 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65c58c88f674c8ea16946629d0ea461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.371033 2399 tablet_service.cc:1467] Processing CreateTablet for tablet d65c58c88f674c8ea16946629d0ea461 (DEFAULT_TABLE table=test-table [id=949713c262424416bce3377c2d4be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.371111 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65c58c88f674c8ea16946629d0ea461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.372212 2714 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Bootstrap starting.
I20260619 20:55:10.372205 2715 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: Bootstrap starting.
I20260619 20:55:10.372294 2107 catalog_manager.cc:4841] TS 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195): tablet a9f90fa9b35a4b8e81c63d74783144a0 (table test-table [id=014883fce9db4ecd97e525de1d77a8cc]) successfully deleted
I20260619 20:55:10.372510 2714 tablet_bootstrap.cc:654] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.372591 2107 catalog_manager.cc:4841] TS b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855): tablet a9f90fa9b35a4b8e81c63d74783144a0 (table test-table [id=014883fce9db4ecd97e525de1d77a8cc]) successfully deleted
I20260619 20:55:10.372632 2715 tablet_bootstrap.cc:654] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.372742 2540 tablet_service.cc:1467] Processing CreateTablet for tablet d65c58c88f674c8ea16946629d0ea461 (DEFAULT_TABLE table=test-table [id=949713c262424416bce3377c2d4be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:10.372866 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65c58c88f674c8ea16946629d0ea461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:10.373208 2715 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: No bootstrap required, opened a new log
I20260619 20:55:10.373246 2715 ts_tablet_manager.cc:1397] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:10.373340 2715 raft_consensus.cc:348] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.373381 2715 raft_consensus.cc:374] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.373394 2715 raft_consensus.cc:729] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.373428 2715 consensus_queue.cc:260] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.373523 2715 ts_tablet_manager.cc:1428] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:10.373936 2716 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Bootstrap starting.
I20260619 20:55:10.374505 2716 tablet_bootstrap.cc:654] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:10.374603 2714 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: No bootstrap required, opened a new log
I20260619 20:55:10.374676 2714 ts_tablet_manager.cc:1397] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:10.374773 2714 raft_consensus.cc:348] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.374868 2714 raft_consensus.cc:374] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.374897 2714 raft_consensus.cc:729] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec0a26d95ff4e32bc5006be8a154024, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.374946 2714 consensus_queue.cc:260] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.375036 2714 ts_tablet_manager.cc:1428] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:10.375140 2716 tablet_bootstrap.cc:492] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: No bootstrap required, opened a new log
I20260619 20:55:10.375190 2716 ts_tablet_manager.cc:1397] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:10.375329 2716 raft_consensus.cc:348] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.375394 2716 raft_consensus.cc:374] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:10.375425 2716 raft_consensus.cc:729] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f375e4b8144051876a866e027cc56e, State: Initialized, Role: FOLLOWER
I20260619 20:55:10.375461 2716 consensus_queue.cc:260] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.375548 2716 ts_tablet_manager.cc:1428] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:10.504005 2720 raft_consensus.cc:482] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:10.504096 2720 raft_consensus.cc:504] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.504464 2720 leader_election.cc:288] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:10.507465 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.504799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59779 (local address 127.0.98.2:46407)
0619 20:55:10.504847 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.504849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.504884 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:55:10.504990 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.504991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.505001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.505073 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:10.505232 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.505697 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.506756 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.507117 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.507202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.507213 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.507362 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.507397 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:55:10.507500 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.504665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46407 (local address 127.0.98.1:59779)
0619 20:55:10.504811 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:10.504825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:10.504896 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:10.505079 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:10.505080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:10.505082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:10.505157 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.505159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.505781 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.505782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.506663 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.506665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.507212 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.507213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.507283 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.507286 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:10.507356 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:55:10.507394 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260619 20:55:10.507627 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.504665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.1:41345)
0619 20:55:10.504919 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:10.504936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:10.505028 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:10.505184 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:10.505185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:10.505187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:10.505262 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.505263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.505962 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.505963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.506773 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.506776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.507357 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.507358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.507432 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.507436 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:10.507503 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:55:10.507539 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260619 20:55:10.507625 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.504767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41345 (local address 127.0.98.3:37195)
0619 20:55:10.504878 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.504880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.505014 (+ 134us) server_negotiation.cc:407] Connection header received
0619 20:55:10.505096 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.505097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.505106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.505168 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:10.505340 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.505859 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.506865 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.507256 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.507354 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.507364 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.507521 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.507558 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260619 20:55:10.507871 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65c58c88f674c8ea16946629d0ea461" candidate_uuid: "b6621e97316e4b4084545187b8f1310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec0a26d95ff4e32bc5006be8a154024" is_pre_election: true
I20260619 20:55:10.508041 2420 raft_consensus.cc:2454] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6621e97316e4b4084545187b8f1310a in term 0.
I20260619 20:55:10.507990 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65c58c88f674c8ea16946629d0ea461" candidate_uuid: "b6621e97316e4b4084545187b8f1310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e" is_pre_election: true
I20260619 20:55:10.508092 2560 raft_consensus.cc:2454] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6621e97316e4b4084545187b8f1310a in term 0.
I20260619 20:55:10.508232 2214 leader_election.cc:302] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec0a26d95ff4e32bc5006be8a154024, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:10.508391 2720 raft_consensus.cc:2781] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:10.508431 2720 raft_consensus.cc:482] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:10.508453 2720 raft_consensus.cc:3037] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.509084 2720 raft_consensus.cc:504] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.509225 2720 leader_election.cc:288] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [CANDIDATE]: Term 1 election: Requested vote from peers 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407), 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195)
I20260619 20:55:10.509368 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65c58c88f674c8ea16946629d0ea461" candidate_uuid: "b6621e97316e4b4084545187b8f1310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec0a26d95ff4e32bc5006be8a154024"
I20260619 20:55:10.509369 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65c58c88f674c8ea16946629d0ea461" candidate_uuid: "b6621e97316e4b4084545187b8f1310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f375e4b8144051876a866e027cc56e"
I20260619 20:55:10.509444 2420 raft_consensus.cc:3037] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.509444 2560 raft_consensus.cc:3037] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:10.510150 2560 raft_consensus.cc:2454] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6621e97316e4b4084545187b8f1310a in term 1.
I20260619 20:55:10.510258 2420 raft_consensus.cc:2454] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6621e97316e4b4084545187b8f1310a in term 1.
I20260619 20:55:10.510334 2211 leader_election.cc:302] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f375e4b8144051876a866e027cc56e, b6621e97316e4b4084545187b8f1310a; no voters:
I20260619 20:55:10.510442 2720 raft_consensus.cc:2781] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:10.510596 2720 raft_consensus.cc:686] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [term 1 LEADER]: Becoming Leader. State: Replica: b6621e97316e4b4084545187b8f1310a, State: Running, Role: LEADER
I20260619 20:55:10.510684 2720 consensus_queue.cc:237] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } }
I20260619 20:55:10.511464 2123 catalog_manager.cc:5495] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a reported cstate change: term changed from 0 to 1, leader changed from <none> to b6621e97316e4b4084545187b8f1310a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b6621e97316e4b4084545187b8f1310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6621e97316e4b4084545187b8f1310a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:10.520450 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.517932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.98.1:39855)
0619 20:55:10.518095 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.518097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.518103 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:10.518141 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.518142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.518159 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.518286 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:10.518404 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.518890 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.519616 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.519842 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.519940 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.520078 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.520202 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.520287 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.520338 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.520358 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260619 20:55:10.524919 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.522684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46407 (local address 127.0.98.1:60821)
0619 20:55:10.522862 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:10.522871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:10.522921 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:10.523021 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:10.523021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:10.523022 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:10.523065 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.523066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.523678 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.523678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.524215 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.524216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.524553 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.524554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.524593 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.524604 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.524792 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.524793 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:10.524827 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:55:10.524858 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260619 20:55:10.524912 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.522721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60821 (local address 127.0.98.2:46407)
0619 20:55:10.522787 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.522789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.522937 (+ 148us) server_negotiation.cc:407] Connection header received
0619 20:55:10.522957 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.522958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.522968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.523017 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:10.523119 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.523614 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.524292 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.524471 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.524522 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.524706 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.524729 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.524776 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.524846 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.524864 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:55:10.526994 2560 raft_consensus.cc:1261] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Refusing update from remote peer b6621e97316e4b4084545187b8f1310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:10.527024 2420 raft_consensus.cc:1261] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Refusing update from remote peer b6621e97316e4b4084545187b8f1310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:10.527211 2720 consensus_queue.cc:1035] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f375e4b8144051876a866e027cc56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:10.527493 2734 consensus_queue.cc:1035] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec0a26d95ff4e32bc5006be8a154024" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:10.534408 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.531935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40007 (local address 127.0.98.1:39855)
0619 20:55:10.531973 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:10.531975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:10.532081 (+ 106us) server_negotiation.cc:407] Connection header received
0619 20:55:10.532155 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:10.532156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:10.532163 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:10.532205 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:10.532281 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.532706 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.533819 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.534068 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.534159 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.534233 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.534261 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.534311 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:10.534345 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 20:55:10.534360 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:10.534425 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:10.531873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.98.2:40007)
0619 20:55:10.532030 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:10.532041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:10.532097 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:10.532213 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:10.532214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:10.532215 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:10.532247 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.532248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.532856 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.532856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.533754 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:10.533756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:10.534125 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:10.534125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:10.534172 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:10.534175 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:10.534312 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:10.534313 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:10.534342 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:55:10.534373 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
W20260619 20:55:10.534782 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:55:12.057920 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:12.058036 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:12.058949 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deaefb012cf846609931edb8a5959b14: Sending DeleteTablet for 3 replicas of tablet d65c58c88f674c8ea16946629d0ea461
I20260619 20:55:12.059211 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet d65c58c88f674c8ea16946629d0ea461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34986
I20260619 20:55:12.059216 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet d65c58c88f674c8ea16946629d0ea461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50004
I20260619 20:55:12.059216 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet d65c58c88f674c8ea16946629d0ea461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44164
I20260619 20:55:12.059418 2767 tablet_replica.cc:330] T d65c58c88f674c8ea16946629d0ea461 P b6621e97316e4b4084545187b8f1310a: stopping tablet replica
I20260619 20:55:12.059463 2766 tablet_replica.cc:330] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: stopping tablet replica
I20260619 20:55:12.059567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2195
I20260619 20:55:12.060241 2766 raft_consensus.cc:2229] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:12.060295 2766 raft_consensus.cc:2258] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:12.060552 2766 ts_tablet_manager.cc:1905] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:12.061992 2768 tablet_replica.cc:330] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: stopping tablet replica
I20260619 20:55:12.062080 2768 raft_consensus.cc:2229] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:12.062150 2768 raft_consensus.cc:2258] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:12.062325 2766 ts_tablet_manager.cc:1918] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:55:12.062398 2766 log.cc:1196] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/d65c58c88f674c8ea16946629d0ea461
I20260619 20:55:12.062439 2768 ts_tablet_manager.cc:1905] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:12.062672 2766 ts_tablet_manager.cc:1939] T d65c58c88f674c8ea16946629d0ea461 P 04f375e4b8144051876a866e027cc56e: Deleting consensus metadata
I20260619 20:55:12.062981 2107 catalog_manager.cc:4841] TS 04f375e4b8144051876a866e027cc56e (127.0.98.3:37195): tablet d65c58c88f674c8ea16946629d0ea461 (table test-table [id=949713c262424416bce3377c2d4be6cb]) successfully deleted
I20260619 20:55:12.064356 2768 ts_tablet_manager.cc:1918] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:55:12.064419 2768 log.cc:1196] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/d65c58c88f674c8ea16946629d0ea461
I20260619 20:55:12.064610 2768 ts_tablet_manager.cc:1939] T d65c58c88f674c8ea16946629d0ea461 P 5ec0a26d95ff4e32bc5006be8a154024: Deleting consensus metadata
I20260619 20:55:12.065047 2110 catalog_manager.cc:4841] TS 5ec0a26d95ff4e32bc5006be8a154024 (127.0.98.2:46407): tablet d65c58c88f674c8ea16946629d0ea461 (table test-table [id=949713c262424416bce3377c2d4be6cb]) successfully deleted
W20260619 20:55:12.066540 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39855 (error 108)
I20260619 20:55:12.066795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2335
I20260619 20:55:12.067580 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:12.066897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39855 (local address 127.0.0.1:35008)
0619 20:55:12.067497 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:12.067542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":51,"threads_started":1}
W20260619 20:55:12.067730 2107 catalog_manager.cc:4584] TS b6621e97316e4b4084545187b8f1310a (127.0.98.1:39855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d65c58c88f674c8ea16946629d0ea461: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39855: connect: Connection refused (error 111)
I20260619 20:55:12.073414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2475
I20260619 20:55:12.077903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2091
2026-06-19T20:55:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6376 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:12 dist-test-slave-8bgz krb5kdc[2773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:12 dist-test-slave-8bgz krb5kdc[2773](info): set up 2 sockets
Jun 19 20:55:12 dist-test-slave-8bgz krb5kdc[2773](info): commencing operation
krb5kdc: starting...
W20260619 20:55:13.109938 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:13.222929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:13.302069 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:13.302376 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:13.302434 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:13.303583 2786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:13.303655 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:13.303679 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:13.303719 2786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:13.303743 2786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:13.305205 2786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:13.305490 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:13.305733 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:13.310742 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.310811 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.310811 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.310799 2786 server_base.cc:1034] running on GCE node
I20260619 20:55:13.311184 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:13.311472 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:13.312644 2786 hybrid_clock.cc:648] HybridClock initialized: now 1781902513312637 us; error 37 us; skew 500 ppm
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:13.315148 2786 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:13.315622 2786 webserver.cc:460] Webserver started at http://127.0.98.62:33791/ using document root <none> and password file <none>
I20260619 20:55:13.315855 2786 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:13.315930 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:13.316080 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:13.317090 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "864c3ea28c9c4e0aab4327e16aa0da4e"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "b87ec9f9bda7b29bf1d2bce0349f9d4d"
server_key_iv: "6e7c9f1d5d1dceec822f98b4cf0c9feb"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.317436 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "864c3ea28c9c4e0aab4327e16aa0da4e"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "b87ec9f9bda7b29bf1d2bce0349f9d4d"
server_key_iv: "6e7c9f1d5d1dceec822f98b4cf0c9feb"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.318813 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:13.319523 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.319695 2786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:13.319795 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "864c3ea28c9c4e0aab4327e16aa0da4e"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "b87ec9f9bda7b29bf1d2bce0349f9d4d"
server_key_iv: "6e7c9f1d5d1dceec822f98b4cf0c9feb"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.319875 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:13.329563 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:13.331822 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:13.331987 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:13.335917 2786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38305
I20260619 20:55:13.335944 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38305 every 8 connection(s)
I20260619 20:55:13.336431 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:13.337059 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:13.337205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2786
I20260619 20:55:13.337316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:13.337482 392 external_mini_cluster.cc:1428] Setting key 9254e3d3978d98b1dbf896ca1eb5b767
I20260619 20:55:13.340391 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e: Bootstrap starting.
I20260619 20:55:13.341118 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:13.341415 2854 log.cc:824] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:13.342098 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e: No bootstrap required, opened a new log
I20260619 20:55:13.344084 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } }
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:13.344251 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:13.344290 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864c3ea28c9c4e0aab4327e16aa0da4e, State: Initialized, Role: FOLLOWER
I20260619 20:55:13.344425 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } }
I20260619 20:55:13.344525 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:13.344558 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:13.344609 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:13.345234 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } }
I20260619 20:55:13.345355 2854 leader_election.cc:302] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864c3ea28c9c4e0aab4327e16aa0da4e; no voters:
I20260619 20:55:13.345527 2854 leader_election.cc:288] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:13.345588 2859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:13.345774 2859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [term 1 LEADER]: Becoming Leader. State: Replica: 864c3ea28c9c4e0aab4327e16aa0da4e, State: Running, Role: LEADER
I20260619 20:55:13.345803 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:13.345871 2859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } }
I20260619 20:55:13.346210 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } } }
I20260619 20:55:13.346392 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:13.346539 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864c3ea28c9c4e0aab4327e16aa0da4e. Latest consensus state: current_term: 1 leader_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864c3ea28c9c4e0aab4327e16aa0da4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38305 } } }
I20260619 20:55:13.346618 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:13.346729 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:13.347684 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:13.349049 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.337956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.98.62:38305)
0619 20:55:13.338106 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:13.338114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:13.338138 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:55:13.338726 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:13.338733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:13.338780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:13.338889 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:13.339664 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.340591 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.341651 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.342137 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.342293 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.344793 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:13.344797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:13.344797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:13.344807 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:13.347091 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.347481 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.347484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.347484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.347589 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.347840 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.347841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.347841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.348159 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:13.348716 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:13.348801 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:55:13.348897 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260619 20:55:13.349751 2868 catalog_manager.cc:1347] Generated new cluster ID: 4a65cabf9e704b4cb3143239a1e3b49f
I20260619 20:55:13.349802 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:13.372215 2868 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:13.372970 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:13.376163 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864c3ea28c9c4e0aab4327e16aa0da4e: Generated new TSK 0
I20260619 20:55:13.376332 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:13.401048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:13.480556 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:13.480748 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:13.480939 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:13.482263 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:13.482314 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:13.482365 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:13.483886 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:13.484125 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:13.484416 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:13.489557 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.489634 2880 server_base.cc:1034] running on GCE node
W20260619 20:55:13.489557 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.489604 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.489939 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:13.490182 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:13.491434 2880 hybrid_clock.cc:648] HybridClock initialized: now 1781902513491428 us; error 36 us; skew 500 ppm
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:13.493777 2880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:13.494310 2880 webserver.cc:460] Webserver started at http://127.0.98.1:35319/ using document root <none> and password file <none>
I20260619 20:55:13.494547 2880 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:13.494613 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:13.494741 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:13.495807 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "2bbd422f33484c49839eb6675f58b12b"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "390bb25f3be3cf30a5e9b96b07a72add"
server_key_iv: "ecfcdd0b270a967828fd396d5d20c136"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.496165 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2bbd422f33484c49839eb6675f58b12b"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "390bb25f3be3cf30a5e9b96b07a72add"
server_key_iv: "ecfcdd0b270a967828fd396d5d20c136"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.497534 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:13.498325 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.498488 2880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:13.498569 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "2bbd422f33484c49839eb6675f58b12b"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "390bb25f3be3cf30a5e9b96b07a72add"
server_key_iv: "ecfcdd0b270a967828fd396d5d20c136"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.498620 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:13.508191 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:13.509827 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:13.509931 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:13.510097 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:13.510426 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:13.510459 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.510479 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:13.510491 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.516309 2880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35669
I20260619 20:55:13.516369 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35669 every 8 connection(s)
I20260619 20:55:13.516671 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:13.525216 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.517403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45291 (local address 127.0.98.62:38305)
0619 20:55:13.517470 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:13.517472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:13.517643 (+ 171us) server_negotiation.cc:407] Connection header received
0619 20:55:13.518570 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:13.518571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:13.518580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:13.518635 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:13.519153 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.519633 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.520523 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.520701 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.520748 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.522799 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:13.522800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:13.522800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:13.522806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:13.524138 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.524511 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.524512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.524512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.524562 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.524763 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.524764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.524764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.524858 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:13.524915 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:13.525100 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:55:13.525156 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:55:13.525408 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.517350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38305 (local address 127.0.98.1:45291)
0619 20:55:13.517506 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:13.517544 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:13.518485 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:13.518685 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:13.518689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:13.518915 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:13.519086 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.519090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.519710 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.519711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.520454 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.520455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.520769 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.520770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.520862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.521352 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:13.521360 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:13.522725 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:13.524209 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.524214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.524215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.524452 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.524622 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.524623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.524623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.524710 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.524910 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:13.524914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:13.525015 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:55:13.525089 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:55:13.525195 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260619 20:55:13.525537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 2880
I20260619 20:55:13.525617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:13.525810 392 external_mini_cluster.cc:1428] Setting key 1321987511c9e51a8fc393412d8d00f7
I20260619 20:55:13.525933 3009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38305
I20260619 20:55:13.526016 3009 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:13.526173 3009 heartbeater.cc:511] Master 127.0.98.62:38305 requested a full tablet report, sending...
I20260619 20:55:13.526727 2818 ts_manager.cc:194] Registered new tserver with Master: 2bbd422f33484c49839eb6675f58b12b (127.0.98.1:35669)
I20260619 20:55:13.527336 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45291
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:13.549434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:13.629062 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:13.629253 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:13.629276 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:13.630437 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:13.630478 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:13.630504 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:13.632009 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:13.632241 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:13.632468 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:13.637568 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.637599 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.637670 3014 server_base.cc:1034] running on GCE node
W20260619 20:55:13.637611 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.637934 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:13.638118 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:13.639266 3014 hybrid_clock.cc:648] HybridClock initialized: now 1781902513639230 us; error 55 us; skew 500 ppm
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:13.641494 3014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:13.641904 3014 webserver.cc:460] Webserver started at http://127.0.98.2:39605/ using document root <none> and password file <none>
I20260619 20:55:13.642122 3014 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:13.642191 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:13.642362 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:13.643357 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "64d6d2d7ab9f435e8d888fa062925337"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "9b8723a22f5c08173cdf59fabf376ac3"
server_key_iv: "9f2d4f723740be37556fcbfb5c700d23"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.643708 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64d6d2d7ab9f435e8d888fa062925337"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "9b8723a22f5c08173cdf59fabf376ac3"
server_key_iv: "9f2d4f723740be37556fcbfb5c700d23"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.645004 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:13.645747 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.645932 3014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:13.646031 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "64d6d2d7ab9f435e8d888fa062925337"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "9b8723a22f5c08173cdf59fabf376ac3"
server_key_iv: "9f2d4f723740be37556fcbfb5c700d23"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.646118 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:13.656489 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:13.658151 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:13.658344 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:13.658578 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:13.658869 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:13.658926 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.658965 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:13.659004 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.664495 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35237
I20260619 20:55:13.664572 3142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35237 every 8 connection(s)
I20260619 20:55:13.664861 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:13.673781 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.665596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42581 (local address 127.0.98.62:38305)
0619 20:55:13.665641 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:13.665643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:13.665856 (+ 213us) server_negotiation.cc:407] Connection header received
0619 20:55:13.666859 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:13.666860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:13.666871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:13.666924 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:13.667521 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.668019 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.669029 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.669226 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.669312 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.671299 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:13.671301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:13.671301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:13.671308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:13.672427 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.672828 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.672829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.672830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.672890 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.673136 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.673137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.673138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.673249 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:13.673327 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:13.673588 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 20:55:13.673693 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:13.673800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3014
I20260619 20:55:13.673905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:13.673928 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.665556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38305 (local address 127.0.98.2:42581)
0619 20:55:13.665729 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:13.665768 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:13.666749 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:13.666992 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:13.666995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:13.667236 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:13.667405 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.667409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.668113 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.668114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.668900 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.668902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.669320 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.669321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.669441 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.669874 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:13.669881 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:13.671196 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:13.672522 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.672528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.672530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.672737 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.672967 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.672969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.672969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.673066 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.673318 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:13.673323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:13.673451 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 20:55:13.673567 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 20:55:13.673714 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260619 20:55:13.674122 392 external_mini_cluster.cc:1428] Setting key b1ad09880576223d16f573d0951d40e9
I20260619 20:55:13.674378 3143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38305
I20260619 20:55:13.674494 3143 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:13.674661 3143 heartbeater.cc:511] Master 127.0.98.62:38305 requested a full tablet report, sending...
I20260619 20:55:13.675042 2818 ts_manager.cc:194] Registered new tserver with Master: 64d6d2d7ab9f435e8d888fa062925337 (127.0.98.2:35237)
I20260619 20:55:13.675418 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42581
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:13.697175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:13.777173 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:13.777418 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:13.777456 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:13.778987 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:13.779067 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:13.779114 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:13.780674 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:13.780946 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:13.781188 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:13.786268 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.786262 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:13.786262 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:13.786602 3148 server_base.cc:1034] running on GCE node
I20260619 20:55:13.786813 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:13.787057 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:13.788229 3148 hybrid_clock.cc:648] HybridClock initialized: now 1781902513788225 us; error 38 us; skew 500 ppm
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:13.794363 3148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:13.794796 3148 webserver.cc:460] Webserver started at http://127.0.98.3:44507/ using document root <none> and password file <none>
I20260619 20:55:13.795033 3148 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:13.795104 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:13.795238 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:13.796272 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "c957a84f1d1a4efeb148f9258292fdcd"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "495b05cb7c8a7643cb433633863a96db"
server_key_iv: "731f1c6b83b21094c8fdea33d9129a51"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.796654 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c957a84f1d1a4efeb148f9258292fdcd"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "495b05cb7c8a7643cb433633863a96db"
server_key_iv: "731f1c6b83b21094c8fdea33d9129a51"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.797966 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:13.798810 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.799006 3148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:55:13.799104 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "c957a84f1d1a4efeb148f9258292fdcd"
format_stamp: "Formatted at 2026-06-19 20:55:13 on dist-test-slave-8bgz"
server_key: "495b05cb7c8a7643cb433633863a96db"
server_key_iv: "731f1c6b83b21094c8fdea33d9129a51"
server_key_version: "encryptionkey@0"
I20260619 20:55:13.799189 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:13.836946 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:13.839026 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:13.839205 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:13.839447 3148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:13.839753 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:13.839807 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.839845 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:13.839874 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:13.845542 3148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37967
I20260619 20:55:13.845618 3276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37967 every 8 connection(s)
I20260619 20:55:13.845970 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:13.852133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3148
I20260619 20:55:13.852197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:13.852397 392 external_mini_cluster.cc:1428] Setting key 63712fe156a05c69e1691c19ac10bcf1
I20260619 20:55:13.855626 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.846751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52977 (local address 127.0.98.62:38305)
0619 20:55:13.846797 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:13.846800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:13.847041 (+ 241us) server_negotiation.cc:407] Connection header received
0619 20:55:13.848094 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:13.848095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:13.848110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:13.848162 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:13.848809 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.849347 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.850469 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.850669 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.850755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.852815 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:13.852816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:13.852817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:13.852824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:13.854248 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.854667 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.854668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.854668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.854725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:13.854978 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:13.854979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:13.854979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:13.855111 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:13.855184 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:13.855454 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 20:55:13.855531 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:13.855813 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:13.846711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38305 (local address 127.0.98.3:52977)
0619 20:55:13.846905 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:13.846940 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:13.847986 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:13.848232 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:13.848235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:13.848474 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:13.848717 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.848738 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.849446 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.849447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.850374 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:13.850376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:13.850763 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:13.850765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:13.850887 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:13.851296 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:13.851304 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:13.852709 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:13.854345 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.854351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.854353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.854575 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.854791 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:13.854792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:13.854792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:13.854898 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:13.855186 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:13.855192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:13.855326 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 20:55:13.855442 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 20:55:13.855594 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260619 20:55:13.856283 3277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38305
I20260619 20:55:13.856360 3277 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:13.856488 3277 heartbeater.cc:511] Master 127.0.98.62:38305 requested a full tablet report, sending...
I20260619 20:55:13.856813 2818 ts_manager.cc:194] Registered new tserver with Master: c957a84f1d1a4efeb148f9258292fdcd (127.0.98.3:37967)
I20260619 20:55:13.857185 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52977
I20260619 20:55:13.863819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:55:13.873847 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:13.873092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.0.98.62:38305)
0619 20:55:13.873144 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:13.873159 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:13.873274 (+ 115us) server_negotiation.cc:407] Connection header received
0619 20:55:13.873309 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:13.873310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:13.873317 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:13.873366 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:13.873804 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55538: BlockingRecv error: recv got EOF from 127.0.0.1:55538 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:55:13.874780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2880
I20260619 20:55:13.879058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3014
I20260619 20:55:13.883226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3148
I20260619 20:55:13.887385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 2786
2026-06-19T20:55:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1808 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[3289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[3289](info): set up 2 sockets
Jun 19 20:55:13 dist-test-slave-8bgz krb5kdc[3289](info): commencing operation
krb5kdc: starting...
W20260619 20:55:14.922102 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:15.036942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:15.118901 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:15.119091 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:15.119112 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:15.120249 3302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:15.120288 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:15.120302 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:15.120316 3302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:15.120370 3302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:15.121794 3302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:15.122036 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:15.122277 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:15.127028 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:15.127043 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:15.127146 3302 server_base.cc:1034] running on GCE node
W20260619 20:55:15.127063 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:15.127588 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:15.127828 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:15.129000 3302 hybrid_clock.cc:648] HybridClock initialized: now 1781902515128983 us; error 28 us; skew 500 ppm
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:15.131757 3302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:15.132217 3302 webserver.cc:460] Webserver started at http://127.0.98.62:46735/ using document root <none> and password file <none>
I20260619 20:55:15.132442 3302 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:15.132509 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:15.132668 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:15.133692 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "75977ef10b2f411b92ad4c78e83c9fc3"
format_stamp: "Formatted at 2026-06-19 20:55:15 on dist-test-slave-8bgz"
server_key: "8175b276de6e0db3a6e2bb603387e176"
server_key_iv: "82650da79881e9af8a81d06af8d299ee"
server_key_version: "encryptionkey@0"
I20260619 20:55:15.134055 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "75977ef10b2f411b92ad4c78e83c9fc3"
format_stamp: "Formatted at 2026-06-19 20:55:15 on dist-test-slave-8bgz"
server_key: "8175b276de6e0db3a6e2bb603387e176"
server_key_iv: "82650da79881e9af8a81d06af8d299ee"
server_key_version: "encryptionkey@0"
I20260619 20:55:15.135349 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:15.136094 3317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:15.136262 3302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:15.136363 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "75977ef10b2f411b92ad4c78e83c9fc3"
format_stamp: "Formatted at 2026-06-19 20:55:15 on dist-test-slave-8bgz"
server_key: "8175b276de6e0db3a6e2bb603387e176"
server_key_iv: "82650da79881e9af8a81d06af8d299ee"
server_key_version: "encryptionkey@0"
I20260619 20:55:15.136443 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:15.153093 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:15.155433 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:15.155597 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:15.159605 3302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40377
I20260619 20:55:15.159633 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40377 every 8 connection(s)
I20260619 20:55:15.159967 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:15.160768 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:15.161031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3302
I20260619 20:55:15.161124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:15.161306 392 external_mini_cluster.cc:1428] Setting key ab5f985cf44427998cc8914a19adcb5c
I20260619 20:55:15.164124 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3: Bootstrap starting.
I20260619 20:55:15.164865 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:15.165145 3370 log.cc:824] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:15.165835 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3: No bootstrap required, opened a new log
I20260619 20:55:15.167725 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } }
I20260619 20:55:15.167898 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:15.167932 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75977ef10b2f411b92ad4c78e83c9fc3, State: Initialized, Role: FOLLOWER
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:15.168043 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } }
I20260619 20:55:15.168116 3370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:15.168152 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:15.168218 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:15.168710 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } }
I20260619 20:55:15.168825 3370 leader_election.cc:302] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75977ef10b2f411b92ad4c78e83c9fc3; no voters:
I20260619 20:55:15.169003 3370 leader_election.cc:288] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:15.169035 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:15.169190 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 75977ef10b2f411b92ad4c78e83c9fc3, State: Running, Role: LEADER
I20260619 20:55:15.169281 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:15.169375 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } }
I20260619 20:55:15.169750 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75977ef10b2f411b92ad4c78e83c9fc3. Latest consensus state: current_term: 1 leader_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } } }
I20260619 20:55:15.169891 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:15.170358 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:15.170475 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75977ef10b2f411b92ad4c78e83c9fc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40377 } } }
I20260619 20:55:15.170640 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:15.170819 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:15.172346 3384 catalog_manager.cc:1347] Generated new cluster ID: f243bc346f4340bd96636243bc12291a
I20260619 20:55:15.172389 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:15.172793 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:15.161800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.98.62:40377)
0619 20:55:15.161975 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:15.161984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:15.162015 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:55:15.162515 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:15.162522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:15.162569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:15.162682 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:15.163431 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:15.164368 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:15.165456 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:15.165955 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:15.166083 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:15.168578 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:15.168582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:15.168582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:15.168593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:15.171308 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:15.171712 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:15.171715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:15.171715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:15.171800 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:15.172012 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:15.172013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:15.172014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:15.172241 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:15.172405 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:15.172509 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:55:15.172636 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260619 20:55:15.173692 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:55:15.194535 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:15.187176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.62:40377)
0619 20:55:15.187218 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:15.187220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:15.190449 (+ 3229us) server_negotiation.cc:407] Connection header received
0619 20:55:15.190517 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:15.190518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:15.190529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:15.190590 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:15.191030 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:15.191776 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:15.192858 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:15.193153 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:15.193219 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:15.193964 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:15.193966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:15.193966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:15.194176 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:15.194201 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:15.194226 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:15.194297 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:15.194399 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:55:15.194429 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":20}
I20260619 20:55:15.206353 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:15.206987 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:15.212061 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75977ef10b2f411b92ad4c78e83c9fc3: Generated new TSK 0
I20260619 20:55:15.212287 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:15.220397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3302
2026-06-19T20:55:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3403](info): set up 2 sockets
Jun 19 20:55:15 dist-test-slave-8bgz krb5kdc[3403](info): commencing operation
krb5kdc: starting...
W20260619 20:55:16.270030 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:16.381150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:16.460479 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:16.460655 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:16.460675 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:16.461761 3416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:16.461802 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:16.461814 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:16.461826 3416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:16.461838 3416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:16.463331 3416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:16.463572 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:16.463792 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:55:16.468564 3416 server_base.cc:1034] running on GCE node
W20260619 20:55:16.468537 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.468545 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.468545 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:16.469017 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:16.469321 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:16.470494 3416 hybrid_clock.cc:648] HybridClock initialized: now 1781902516470480 us; error 25 us; skew 500 ppm
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:16.472833 3416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:16.473266 3416 webserver.cc:460] Webserver started at http://127.0.98.62:34601/ using document root <none> and password file <none>
I20260619 20:55:16.473500 3416 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:16.473568 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:16.473716 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:16.474745 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "cb37bba2e998d88b6c6cffff156ed8c6"
server_key_iv: "31e63b5ad51803fa5bdc06f475cbf8f6"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.475082 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "cb37bba2e998d88b6c6cffff156ed8c6"
server_key_iv: "31e63b5ad51803fa5bdc06f475cbf8f6"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.476459 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:16.477111 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.477279 3416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:16.477387 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "cb37bba2e998d88b6c6cffff156ed8c6"
server_key_iv: "31e63b5ad51803fa5bdc06f475cbf8f6"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.477460 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:16.504771 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:16.507090 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:16.507251 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:16.511104 3416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42047
I20260619 20:55:16.511137 3483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42047 every 8 connection(s)
I20260619 20:55:16.511492 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:16.512403 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:16.514617 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8: Bootstrap starting.
I20260619 20:55:16.515156 3484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:16.515434 3484 log.cc:824] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:16.516132 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8: No bootstrap required, opened a new log
I20260619 20:55:16.516638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3416
I20260619 20:55:16.516738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:16.516924 392 external_mini_cluster.cc:1428] Setting key e11d9188c3b2f2a14646d5d53f44f2ec
I20260619 20:55:16.518318 3484 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260619 20:55:16.518476 3484 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:16.518534 3484 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb53f0305fd645bfbc6a26c46ec4bfb8, State: Initialized, Role: FOLLOWER
I20260619 20:55:16.518661 3484 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260619 20:55:16.518762 3484 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:16.518797 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:16.518841 3484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:16.519572 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260619 20:55:16.519717 3484 leader_election.cc:302] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb53f0305fd645bfbc6a26c46ec4bfb8; no voters:
I20260619 20:55:16.519852 3484 leader_election.cc:288] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:16.519878 3489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:16.519970 3489 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [term 1 LEADER]: Becoming Leader. State: Replica: eb53f0305fd645bfbc6a26c46ec4bfb8, State: Running, Role: LEADER
I20260619 20:55:16.520078 3484 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:16.520092 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260619 20:55:16.520377 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb53f0305fd645bfbc6a26c46ec4bfb8. Latest consensus state: current_term: 1 leader_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } } }
I20260619 20:55:16.520478 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:16.520687 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:16.521329 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:16.521899 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb53f0305fd645bfbc6a26c46ec4bfb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } } }
I20260619 20:55:16.522018 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:16.523279 3496 catalog_manager.cc:1347] Generated new cluster ID: 7e22241381db4a0f82ecdc5bf974e7bf
I20260619 20:55:16.523329 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:16.528120 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.517401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.0.98.62:42047)
0619 20:55:16.517621 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:16.517630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:16.517661 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:55:16.518303 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:16.518310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:16.518356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:16.518488 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:16.519181 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.520214 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.521216 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.521779 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.521875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:16.524670 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:16.524673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:16.524673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:16.524683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:16.526492 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.526801 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.526804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.526804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.526903 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.527049 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.527049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.527049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.527279 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:16.527425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:16.527818 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 20:55:16.527942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260619 20:55:16.530974 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:16.531436 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:16.542107 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb53f0305fd645bfbc6a26c46ec4bfb8: Generated new TSK 0
I20260619 20:55:16.542304 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:16.566346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:16.645465 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:16.645653 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:16.645673 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:16.646858 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:16.646898 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:16.646924 3510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:16.648212 3510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:16.648370 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:16.648532 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:16.653463 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.653463 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.653507 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:16.653715 3510 server_base.cc:1034] running on GCE node
I20260619 20:55:16.653901 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:16.654084 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:16.655253 3510 hybrid_clock.cc:648] HybridClock initialized: now 1781902516655237 us; error 26 us; skew 500 ppm
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:16.657760 3510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:16.658273 3510 webserver.cc:460] Webserver started at http://127.0.98.1:34213/ using document root <none> and password file <none>
I20260619 20:55:16.658488 3510 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:16.658572 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:16.658727 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:16.659735 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d09fd70fe6443ffb7802fe6e09c6e0a"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "eea2dbe2a3038f773163739b1a341296"
server_key_iv: "3bf2f677c965e1c226007bc406b3628a"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.660089 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d09fd70fe6443ffb7802fe6e09c6e0a"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "eea2dbe2a3038f773163739b1a341296"
server_key_iv: "3bf2f677c965e1c226007bc406b3628a"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.661337 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:16.662042 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.662225 3510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:16.662309 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "7d09fd70fe6443ffb7802fe6e09c6e0a"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "eea2dbe2a3038f773163739b1a341296"
server_key_iv: "3bf2f677c965e1c226007bc406b3628a"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.662395 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:16.697307 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:16.699285 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:16.699460 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:16.699712 3510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:16.700018 3510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:16.700073 3510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.700109 3510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:16.700135 3510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.705610 3510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36297
I20260619 20:55:16.705664 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36297 every 8 connection(s)
I20260619 20:55:16.706027 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:16.710776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3510
I20260619 20:55:16.710887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:16.711097 392 external_mini_cluster.cc:1428] Setting key c488f1c88929a55d1b4959b1301e38bc
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:16.715543 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.706829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36275 (local address 127.0.98.62:42047)
0619 20:55:16.706876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:16.706879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:16.707095 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:55:16.707552 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:16.707553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:16.707563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:16.707610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:16.708456 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.708989 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.709937 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.710130 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.710178 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:16.712539 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:16.712540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:16.712541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:16.712550 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:16.714017 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.714566 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.714567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.714568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.714617 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.714892 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.714893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.714893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.714989 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:16.715038 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:16.715401 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 20:55:16.715483 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:16.715794 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.706811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.1:36275)
0619 20:55:16.706971 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:16.707006 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:16.707468 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:16.707678 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:16.707681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:16.708203 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:16.708385 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:16.708390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.709062 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.709063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:16.709861 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:16.709863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.710200 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.710201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:16.710313 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:16.710752 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:16.710763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:16.712443 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:16.714099 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:16.714105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:16.714107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:16.714462 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:16.714683 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:16.714684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:16.714684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:16.714823 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:16.715069 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:16.715075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:16.715253 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 20:55:16.715379 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 20:55:16.715568 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260619 20:55:16.716367 3639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260619 20:55:16.716454 3639 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:16.716620 3639 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260619 20:55:16.717104 3448 ts_manager.cc:194] Registered new tserver with Master: 7d09fd70fe6443ffb7802fe6e09c6e0a (127.0.98.1:36297)
I20260619 20:55:16.717628 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36275
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:16.735214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:16.813403 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:16.813632 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:16.813673 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:16.814896 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:16.814975 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:16.815021 3644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:16.816422 3644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:16.816684 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:16.816907 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:16.821873 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.821902 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.821967 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:16.822104 3644 server_base.cc:1034] running on GCE node
I20260619 20:55:16.822350 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:16.822541 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:16.823705 3644 hybrid_clock.cc:648] HybridClock initialized: now 1781902516823704 us; error 47 us; skew 500 ppm
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:16.826135 3644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:16.826613 3644 webserver.cc:460] Webserver started at http://127.0.98.2:42093/ using document root <none> and password file <none>
I20260619 20:55:16.826851 3644 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:16.826921 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:16.827082 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:16.828084 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e9bd1b92b9a44a297579d112b6fd22b"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "e0b8aa76b884ff1149348bc9a03b55f1"
server_key_iv: "d8482ead19595f962f7b2005dbe2df9c"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.828441 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e9bd1b92b9a44a297579d112b6fd22b"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "e0b8aa76b884ff1149348bc9a03b55f1"
server_key_iv: "d8482ead19595f962f7b2005dbe2df9c"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.829885 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:16.830664 3659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.830839 3644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.830933 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "0e9bd1b92b9a44a297579d112b6fd22b"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "e0b8aa76b884ff1149348bc9a03b55f1"
server_key_iv: "d8482ead19595f962f7b2005dbe2df9c"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.831022 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:16.842973 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:16.845266 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:16.845433 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:16.845662 3644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:16.845964 3644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:16.846020 3644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.846061 3644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:16.846088 3644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.851907 3644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38895
I20260619 20:55:16.851949 3772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38895 every 8 connection(s)
I20260619 20:55:16.852249 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:16.859556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3644
I20260619 20:55:16.859633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:16.859817 392 external_mini_cluster.cc:1428] Setting key ca92805c92aed53b631ea1e38a117fdb
I20260619 20:55:16.860754 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.852938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35111 (local address 127.0.98.62:42047)
0619 20:55:16.853004 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:16.853006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:16.853205 (+ 199us) server_negotiation.cc:407] Connection header received
0619 20:55:16.853681 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:16.853682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:16.853691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:16.853748 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:16.854609 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.855093 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.856009 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.856184 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.856231 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:16.858096 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:16.858097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:16.858097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:16.858104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:16.859345 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.859768 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.859769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.859770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.859832 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:16.860119 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:16.860121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:16.860121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:16.860245 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:16.860306 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:16.860630 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 20:55:16.860696 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:16.861078 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.852916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.2:35111)
0619 20:55:16.853066 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:16.853104 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:16.853600 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:16.853798 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:16.853801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:16.854349 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:16.854535 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:16.854540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.855169 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.855170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:16.855938 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:16.855940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.856251 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:16.856252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:16.856345 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:16.856743 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:16.856750 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:16.858013 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:16.859417 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:16.859422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:16.859423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:16.859681 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:16.859917 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:16.859918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:16.859918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:16.860067 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:16.860308 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:16.860314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:16.860478 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 20:55:16.860609 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 20:55:16.860772 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260619 20:55:16.861545 3773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260619 20:55:16.861639 3773 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:16.861822 3773 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260619 20:55:16.862251 3448 ts_manager.cc:194] Registered new tserver with Master: 0e9bd1b92b9a44a297579d112b6fd22b (127.0.98.2:38895)
I20260619 20:55:16.862632 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35111
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:16.884465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:16.963053 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:16.963287 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:16.963328 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:16.964465 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:16.964533 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:16.964581 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:16.966099 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:16.966391 3778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:16.966639 3778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:16.972007 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.972002 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:16.972002 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:16.972342 3778 server_base.cc:1034] running on GCE node
I20260619 20:55:16.972543 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:16.972787 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:16.973963 3778 hybrid_clock.cc:648] HybridClock initialized: now 1781902516973948 us; error 22 us; skew 500 ppm
Jun 19 20:55:16 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:16.976449 3778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:16.976909 3778 webserver.cc:460] Webserver started at http://127.0.98.3:42181/ using document root <none> and password file <none>
I20260619 20:55:16.977181 3778 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:16.977267 3778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:16.977406 3778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:16.978478 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7e69b98e54642fc8c7195e5acaf0bbb"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "691bca88c3f5392d521fe4dad4892853"
server_key_iv: "7e028838d9705444cf4af69bc9dbeffc"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.978839 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7e69b98e54642fc8c7195e5acaf0bbb"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "691bca88c3f5392d521fe4dad4892853"
server_key_iv: "7e028838d9705444cf4af69bc9dbeffc"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.980185 3778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:16.980978 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.981184 3778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:16.981266 3778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "d7e69b98e54642fc8c7195e5acaf0bbb"
format_stamp: "Formatted at 2026-06-19 20:55:16 on dist-test-slave-8bgz"
server_key: "691bca88c3f5392d521fe4dad4892853"
server_key_iv: "7e028838d9705444cf4af69bc9dbeffc"
server_key_version: "encryptionkey@0"
I20260619 20:55:16.981348 3778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:16.987854 3778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:16.989776 3778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:16.989928 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:16.990139 3778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:16.990454 3778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:16.990494 3778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.990514 3778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:16.990563 3778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:16.996444 3778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40871
I20260619 20:55:16.996498 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40871 every 8 connection(s)
I20260619 20:55:16.996773 3778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:16.998677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3778
I20260619 20:55:16.998785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:16.998993 392 external_mini_cluster.cc:1428] Setting key 4331e0a2e9df13077835cef0fea30279
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:17.005699 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.997422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50085 (local address 127.0.98.62:42047)
0619 20:55:16.997500 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:16.997502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:16.997688 (+ 186us) server_negotiation.cc:407] Connection header received
0619 20:55:16.998183 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:16.998184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:16.998196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:16.998274 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:16.999467 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:16.999963 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.000955 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:17.001150 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.001240 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:17.003209 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:17.003211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:17.003211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:17.003219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:17.004366 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:17.004763 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:17.004765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:17.004765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:17.004826 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:17.005069 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:17.005070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:17.005070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:17.005190 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:17.005267 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:17.005530 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 20:55:17.005609 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 20:55:17.005870 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:16.997367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.3:50085)
0619 20:55:16.997562 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:16.997599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:16.998081 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:16.998340 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:16.998344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:16.999144 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:16.999345 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:16.999350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:17.000059 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.000060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:17.000842 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:17.000844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:17.001267 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.001268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:17.001386 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:17.001804 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:17.001811 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:17.003114 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:17.004469 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:17.004475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:17.004477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:17.004685 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:17.004894 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:17.004895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:17.004896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:17.004999 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:17.005258 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:17.005264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:17.005393 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 20:55:17.005508 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 20:55:17.005652 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260619 20:55:17.006361 3907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260619 20:55:17.006439 3907 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:17.006620 3907 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260619 20:55:17.006985 3448 ts_manager.cc:194] Registered new tserver with Master: d7e69b98e54642fc8c7195e5acaf0bbb (127.0.98.3:40871)
I20260619 20:55:17.007342 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50085
I20260619 20:55:17.010604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902517, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902517, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:17.031800 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:17.025216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.98.62:42047)
0619 20:55:17.025269 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:17.025271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:17.025388 (+ 117us) server_negotiation.cc:407] Connection header received
0619 20:55:17.025435 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:17.025435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:17.025444 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:17.025500 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:17.026025 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:17.026580 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.027303 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:17.027481 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:17.027536 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:17.029697 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:17.029698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:17.029698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:17.029704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:17.030918 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:17.031208 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:17.031209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:17.031209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:17.031255 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:17.031412 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:17.031412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:17.031413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:17.031533 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:17.031605 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:17.031690 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:55:17.031741 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 20:55:17.032004 3448 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37490
I20260619 20:55:17.032502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3510
I20260619 20:55:17.037088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3644
I20260619 20:55:17.041354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3778
I20260619 20:55:17.045581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3416
2026-06-19T20:55:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1827 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3919](info): set up 2 sockets
Jun 19 20:55:17 dist-test-slave-8bgz krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260619 20:55:18.078083 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:18.188741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:18.266172 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:18.266480 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:18.266515 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:18.267876 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:18.267985 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:18.268029 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:18.268064 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:18.268085 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:18.269811 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:18.270073 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:18.270352 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:18.275084 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.275099 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.275104 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.275167 3932 server_base.cc:1034] running on GCE node
I20260619 20:55:18.275522 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:18.275837 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:18.276999 3932 hybrid_clock.cc:648] HybridClock initialized: now 1781902518277013 us; error 51 us; skew 500 ppm
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:18.279352 3932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:18.279750 3932 webserver.cc:460] Webserver started at http://127.0.98.62:38925/ using document root <none> and password file <none>
I20260619 20:55:18.279986 3932 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:18.280058 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:18.280215 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:18.281216 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "6ca436e327e54f9bba2d3b82ca16748e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "4cd821e1e84ae9ab4e4ab497896fc1aa"
server_key_iv: "207b997a6c785b19aa4db83ade2e7e9f"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.281563 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ca436e327e54f9bba2d3b82ca16748e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "4cd821e1e84ae9ab4e4ab497896fc1aa"
server_key_iv: "207b997a6c785b19aa4db83ade2e7e9f"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.282944 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:18.283633 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.283797 3932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:18.283896 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "6ca436e327e54f9bba2d3b82ca16748e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "4cd821e1e84ae9ab4e4ab497896fc1aa"
server_key_iv: "207b997a6c785b19aa4db83ade2e7e9f"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.283974 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:18.295513 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:18.297156 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:18.297317 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:18.300957 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40389
I20260619 20:55:18.301015 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40389 every 8 connection(s)
I20260619 20:55:18.301358 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:18.302139 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:18.303337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 3932
I20260619 20:55:18.303452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:18.303633 392 external_mini_cluster.cc:1428] Setting key 66f20bcbc260c38164609ebda345eb80
I20260619 20:55:18.304813 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e: Bootstrap starting.
I20260619 20:55:18.305548 4000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:18.305856 4000 log.cc:824] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:18.306567 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e: No bootstrap required, opened a new log
I20260619 20:55:18.308943 4000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } }
I20260619 20:55:18.309106 4000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:18.309149 4000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca436e327e54f9bba2d3b82ca16748e, State: Initialized, Role: FOLLOWER
I20260619 20:55:18.309279 4000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } }
I20260619 20:55:18.309361 4000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:18.309398 4000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:18.309454 4000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:18.310112 4000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } }
I20260619 20:55:18.310194 4000 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca436e327e54f9bba2d3b82ca16748e; no voters:
I20260619 20:55:18.310391 4000 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:18.310410 4005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:18.310524 4005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [term 1 LEADER]: Becoming Leader. State: Replica: 6ca436e327e54f9bba2d3b82ca16748e, State: Running, Role: LEADER
I20260619 20:55:18.310600 4005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } }
I20260619 20:55:18.310647 4000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:18.310917 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca436e327e54f9bba2d3b82ca16748e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } } }
I20260619 20:55:18.311065 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:18.311259 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca436e327e54f9bba2d3b82ca16748e. Latest consensus state: current_term: 1 leader_uuid: "6ca436e327e54f9bba2d3b82ca16748e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca436e327e54f9bba2d3b82ca16748e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40389 } } }
I20260619 20:55:18.311329 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:18.311429 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:18.312300 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:18.314077 4013 catalog_manager.cc:1347] Generated new cluster ID: 3c060c04aa754ad3a7400079ce580ecf
I20260619 20:55:18.314168 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:18.314579 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.304111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.0.98.62:40389)
0619 20:55:18.304289 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.304297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.304321 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:55:18.304705 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.304711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.304753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.304861 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.305667 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.306789 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.307807 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.308174 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.308247 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.311010 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.311013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.311013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.311024 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.313006 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.313401 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.313404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.313405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.313511 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.313728 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.313730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.313730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.313993 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.314123 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.314257 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.314391 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260619 20:55:18.319427 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:18.320097 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:18.331945 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca436e327e54f9bba2d3b82ca16748e: Generated new TSK 0
I20260619 20:55:18.332098 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:18.360630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:18.452039 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:18.452260 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:18.452299 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:18.453450 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:18.453526 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:18.453572 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:18.455113 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:18.455363 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:18.455606 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:18.460551 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.460553 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.460580 4026 server_base.cc:1034] running on GCE node
W20260619 20:55:18.460554 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.460948 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:18.461189 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:18.462352 4026 hybrid_clock.cc:648] HybridClock initialized: now 1781902518462330 us; error 38 us; skew 500 ppm
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:18.464820 4026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:18.465487 4026 webserver.cc:460] Webserver started at http://127.0.98.1:36827/ using document root <none> and password file <none>
I20260619 20:55:18.465699 4026 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:18.465749 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:18.465833 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:18.466881 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "747b0ad4102c4b04b673bcd58bf45890"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "3a64126aefa2fd6a22c7bae381e3106d"
server_key_iv: "c1d70e79ce054f1eebfa996b861cefca"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.467195 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "747b0ad4102c4b04b673bcd58bf45890"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "3a64126aefa2fd6a22c7bae381e3106d"
server_key_iv: "c1d70e79ce054f1eebfa996b861cefca"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.468372 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:18.468984 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.469161 4026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:18.469242 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "747b0ad4102c4b04b673bcd58bf45890"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "3a64126aefa2fd6a22c7bae381e3106d"
server_key_iv: "c1d70e79ce054f1eebfa996b861cefca"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.469293 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:18.478992 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:18.480830 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:18.480937 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:18.481106 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:18.481412 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:18.481446 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.481467 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:18.481516 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.486927 4026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33541
I20260619 20:55:18.486989 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33541 every 8 connection(s)
I20260619 20:55:18.487349 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:18.496059 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.488145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43339 (local address 127.0.98.62:40389)
0619 20:55:18.488223 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.488226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.488340 (+ 114us) server_negotiation.cc:407] Connection header received
0619 20:55:18.489212 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.489213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.489221 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.489269 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.489837 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.490404 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.491354 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.491547 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.491593 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.493568 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.493570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.493570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.493577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.495007 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.495344 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.495345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.495345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.495392 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.495589 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.495589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.495590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.495679 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.495725 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.495938 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.495998 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 20:55:18.496263 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.488088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40389 (local address 127.0.98.1:43339)
0619 20:55:18.488229 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:18.488262 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:18.489114 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:18.489319 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:18.489322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:18.489543 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:18.489739 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.489743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.490484 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.490485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.491282 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.491284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.491616 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.491617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.491704 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.492152 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:18.492159 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:18.493488 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:18.495072 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.495077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.495078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.495281 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.495449 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.495450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.495450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.495535 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.495732 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:18.495735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:18.495836 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:55:18.495912 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:55:18.496017 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":91,"threads_started":1}
I20260619 20:55:18.496863 4155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40389
I20260619 20:55:18.496958 4155 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:18.497126 4155 heartbeater.cc:511] Master 127.0.98.62:40389 requested a full tablet report, sending...
I20260619 20:55:18.497534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4026
I20260619 20:55:18.497640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:18.497660 3964 ts_manager.cc:194] Registered new tserver with Master: 747b0ad4102c4b04b673bcd58bf45890 (127.0.98.1:33541)
I20260619 20:55:18.497834 392 external_mini_cluster.cc:1428] Setting key 104e3840c588d74008ed90c9abc93a47
I20260619 20:55:18.498438 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43339
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:18.521008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:18.600031 4160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:18.600212 4160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:18.600235 4160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:18.601361 4160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:18.601408 4160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:18.601436 4160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:18.602945 4160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:18.603192 4160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:18.603406 4160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:18.608462 4168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.608501 4165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.608528 4160 server_base.cc:1034] running on GCE node
W20260619 20:55:18.608470 4166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.608994 4160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:18.609241 4160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:18.610391 4160 hybrid_clock.cc:648] HybridClock initialized: now 1781902518610377 us; error 28 us; skew 500 ppm
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:18.613030 4160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:18.613500 4160 webserver.cc:460] Webserver started at http://127.0.98.2:44541/ using document root <none> and password file <none>
I20260619 20:55:18.613739 4160 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:18.613811 4160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:18.613970 4160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:18.615051 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "84d0d49a677744a5905f06279787422e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "ed981a8cc208a2ac29e740982b99c81e"
server_key_iv: "3e312a8b2484d5902fd23abb974768be"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.615449 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84d0d49a677744a5905f06279787422e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "ed981a8cc208a2ac29e740982b99c81e"
server_key_iv: "3e312a8b2484d5902fd23abb974768be"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.616779 4160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:18.617478 4175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.617650 4160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:18.617748 4160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "84d0d49a677744a5905f06279787422e"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "ed981a8cc208a2ac29e740982b99c81e"
server_key_iv: "3e312a8b2484d5902fd23abb974768be"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.617830 4160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:18.631417 4160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:18.633136 4160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:18.633292 4160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:18.633507 4160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:18.633818 4160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:18.633888 4160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.633925 4160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:18.633955 4160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.639400 4160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39853
I20260619 20:55:18.639444 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39853 every 8 connection(s)
I20260619 20:55:18.639726 4160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:18.645280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4160
I20260619 20:55:18.645399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:18.645581 392 external_mini_cluster.cc:1428] Setting key c7b230a6e822888603cd6ab201b3e234
I20260619 20:55:18.648550 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.640359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58961 (local address 127.0.98.62:40389)
0619 20:55:18.640399 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.640401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.640574 (+ 173us) server_negotiation.cc:407] Connection header received
0619 20:55:18.641453 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.641454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.641463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.641519 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.642110 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.642611 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.643553 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.643735 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.643786 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.645811 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.645812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.645812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.645819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.647223 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.647623 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.647625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.647625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.647685 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.647932 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.647933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.647933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.648045 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.648122 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.648383 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.648459 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:18.648744 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.640329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40389 (local address 127.0.98.2:58961)
0619 20:55:18.640472 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:18.640509 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:18.641364 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:18.641591 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:18.641594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:18.641851 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:18.642038 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.642041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.642694 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.642694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.643478 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.643480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.643805 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.643805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.643896 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.644333 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:18.644340 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:18.645699 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:18.647310 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.647316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.647318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.647534 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.647754 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.647755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.647755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.647864 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.648115 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:18.648120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:18.648255 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 20:55:18.648362 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:55:18.648510 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260619 20:55:18.649309 4289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40389
I20260619 20:55:18.649406 4289 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:18.649564 4289 heartbeater.cc:511] Master 127.0.98.62:40389 requested a full tablet report, sending...
I20260619 20:55:18.649994 3964 ts_manager.cc:194] Registered new tserver with Master: 84d0d49a677744a5905f06279787422e (127.0.98.2:39853)
I20260619 20:55:18.650419 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:18.668876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:18.747258 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:18.747442 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:18.747463 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:18.748612 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:18.748652 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:18.748678 4294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:18.750195 4294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:18.750442 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:18.750659 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:18.755630 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.755625 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:18.755605 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:18.755681 4294 server_base.cc:1034] running on GCE node
I20260619 20:55:18.756035 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:18.756230 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:18.757406 4294 hybrid_clock.cc:648] HybridClock initialized: now 1781902518757396 us; error 27 us; skew 500 ppm
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:18.759898 4294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:18.760255 4294 webserver.cc:460] Webserver started at http://127.0.98.3:43023/ using document root <none> and password file <none>
I20260619 20:55:18.760429 4294 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:18.760468 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:18.760550 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:18.761490 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "05ba70c686844cffa3754d868e2f0e25"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "a28baeb3a1407290f4a2b733a4d55881"
server_key_iv: "f78a62b3e497b32666995624782dcd3a"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.761780 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "05ba70c686844cffa3754d868e2f0e25"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "a28baeb3a1407290f4a2b733a4d55881"
server_key_iv: "f78a62b3e497b32666995624782dcd3a"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.763115 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:18.763770 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.763962 4294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:18.764038 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "05ba70c686844cffa3754d868e2f0e25"
format_stamp: "Formatted at 2026-06-19 20:55:18 on dist-test-slave-8bgz"
server_key: "a28baeb3a1407290f4a2b733a4d55881"
server_key_iv: "f78a62b3e497b32666995624782dcd3a"
server_key_version: "encryptionkey@0"
I20260619 20:55:18.764088 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:18.783488 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:18.786182 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:18.786384 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:18.786638 4294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:18.787065 4294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:18.787115 4294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.787151 4294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:18.787170 4294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:18.794456 4294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37045
I20260619 20:55:18.794497 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37045 every 8 connection(s)
I20260619 20:55:18.794888 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:18.803342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4294
I20260619 20:55:18.803438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:18.803660 392 external_mini_cluster.cc:1428] Setting key 88a184998b6a58bade889d198eff72ab
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:18.809403 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.795631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40389 (local address 127.0.98.3:50261)
0619 20:55:18.795787 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:18.795840 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:18.797224 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:18.797607 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:18.797612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:18.797972 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:18.798291 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.798297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.798973 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.798973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.800254 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:18.800258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.800679 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.800680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:18.800805 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.804756 (+ 3951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:18.804766 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:18.806260 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:18.808110 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.808115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.808116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.808335 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.808550 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:18.808551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:18.808551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:18.808642 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:18.808933 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:18.808936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:18.809045 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 20:55:18.809125 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:55:18.809248 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260619 20:55:18.809398 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.795629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50261 (local address 127.0.98.62:40389)
0619 20:55:18.795709 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.795712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.795920 (+ 208us) server_negotiation.cc:407] Connection header received
0619 20:55:18.797367 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.797368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.797378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.797469 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.798373 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.798876 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.800383 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.800587 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.800661 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.806344 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.806346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.806346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.806356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.808029 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.808411 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.808413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.808413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.808480 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.808708 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.808709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.808709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.808862 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.808942 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.809173 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.809269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:55:18.809939 4423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40389
I20260619 20:55:18.810048 4423 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:18.810315 4423 heartbeater.cc:511] Master 127.0.98.62:40389 requested a full tablet report, sending...
I20260619 20:55:18.810783 3964 ts_manager.cc:194] Registered new tserver with Master: 05ba70c686844cffa3754d868e2f0e25 (127.0.98.3:37045)
I20260619 20:55:18.811360 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50261
I20260619 20:55:18.815227 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:18.837610 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.830220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.1:33541)
0619 20:55:18.830381 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.830387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.830399 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:55:18.830480 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.830483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.830498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.830591 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.831152 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.831870 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.832725 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.833036 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.833092 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.835173 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.835178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.835179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.835185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.836743 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.837025 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.837028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.837028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.837078 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.837223 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.837223 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.837224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.837326 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.837422 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.837480 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.837536 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
Jun 19 20:55:18 dist-test-slave-8bgz krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:18.845273 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:18.838790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.98.62:40389)
0619 20:55:18.838855 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:18.838857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:18.838865 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:18.838913 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:18.838914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:18.838922 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:18.838966 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:18.839380 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.839817 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.840403 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:18.840566 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:18.840611 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:18.842777 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:18.842778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:18.842779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:18.842786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:18.844144 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.844497 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.844499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.844499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.844555 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:18.844752 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:18.844753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:18.844754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:18.844892 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:18.844993 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:18.845103 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:55:18.845172 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 20:55:18.845470 3964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49032
I20260619 20:55:18.845816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4026
I20260619 20:55:18.850675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4160
I20260619 20:55:18.855067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4294
I20260619 20:55:18.859368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 3932
2026-06-19T20:55:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1812 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:19 dist-test-slave-8bgz krb5kdc[4435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:19 dist-test-slave-8bgz krb5kdc[4435](info): set up 2 sockets
Jun 19 20:55:19 dist-test-slave-8bgz krb5kdc[4435](info): commencing operation
krb5kdc: starting...
W20260619 20:55:20.386397 392 mini_kdc.cc:121] Time spent starting KDC: real 1.508s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:20.500710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:20.582643 4448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:20.582890 4448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:20.582930 4448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:20.584069 4448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:20.584143 4448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:20.584172 4448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:20.584203 4448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:20.584246 4448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:20.585711 4448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:20.585973 4448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:20.586192 4448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:20.590991 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:20.591128 4448 server_base.cc:1034] running on GCE node
W20260619 20:55:20.590987 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:20.591078 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:20.591554 4448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:20.591811 4448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:20.592969 4448 hybrid_clock.cc:648] HybridClock initialized: now 1781902520592958 us; error 30 us; skew 500 ppm
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:20.595441 4448 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:20.595813 4448 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:20.595888 4448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:20.596035 4448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:20.597076 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "495e28a2553e45188aad1b24f99406f6"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "b3fe6cbfeef8fe80945fe8ed2fd7c4f1"
server_key_iv: "c59b0a1b17b48808e012224ae995fe6e"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.597452 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "495e28a2553e45188aad1b24f99406f6"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "b3fe6cbfeef8fe80945fe8ed2fd7c4f1"
server_key_iv: "c59b0a1b17b48808e012224ae995fe6e"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.598845 4448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:20.599522 4462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.599689 4448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:20.599798 4448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "495e28a2553e45188aad1b24f99406f6"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "b3fe6cbfeef8fe80945fe8ed2fd7c4f1"
server_key_iv: "c59b0a1b17b48808e012224ae995fe6e"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.599885 4448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:20.614075 4448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:20.615855 4448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:20.616016 4448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:20.620128 4448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44049
I20260619 20:55:20.620147 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44049 every 8 connection(s)
I20260619 20:55:20.620429 4448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:20.621268 4515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:20.623409 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: Bootstrap starting.
I20260619 20:55:20.623898 4515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:20.624119 4515 log.cc:824] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:20.624719 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: No bootstrap required, opened a new log
I20260619 20:55:20.624874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4448
I20260619 20:55:20.624984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:20.625185 392 external_mini_cluster.cc:1428] Setting key 99d44695c4d2d4aabe75c2c705fdeedb
I20260619 20:55:20.626638 4515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260619 20:55:20.626760 4515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:20.626807 4515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495e28a2553e45188aad1b24f99406f6, State: Initialized, Role: FOLLOWER
I20260619 20:55:20.626971 4515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260619 20:55:20.627058 4515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:20.627092 4515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:20.627166 4515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:20.627795 4515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260619 20:55:20.627877 4515 leader_election.cc:302] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495e28a2553e45188aad1b24f99406f6; no voters:
I20260619 20:55:20.628072 4515 leader_election.cc:288] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:20.628098 4520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:20.628197 4520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [term 1 LEADER]: Becoming Leader. State: Replica: 495e28a2553e45188aad1b24f99406f6, State: Running, Role: LEADER
I20260619 20:55:20.628273 4520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } }
I20260619 20:55:20.628314 4515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:20.628592 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495e28a2553e45188aad1b24f99406f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } } }
I20260619 20:55:20.628744 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:20.628938 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495e28a2553e45188aad1b24f99406f6. Latest consensus state: current_term: 1 leader_uuid: "495e28a2553e45188aad1b24f99406f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e28a2553e45188aad1b24f99406f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44049 } } }
I20260619 20:55:20.629067 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:20.629350 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:20.629853 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:20.631528 4527 catalog_manager.cc:1347] Generated new cluster ID: 4f24082daa404458adeaa830a8ffd2ff
I20260619 20:55:20.631577 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:20.637182 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:20.625741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.98.62:44049)
0619 20:55:20.625926 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:20.625933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:20.625959 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:55:20.626632 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:20.626638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:20.626684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:20.626839 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:20.627611 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.628878 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.630112 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.630715 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.630848 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:20.633513 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:20.633517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:20.633517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:20.633529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:20.635632 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.636002 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.636005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.636005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.636131 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.636335 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.636336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.636336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.636628 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:20.636797 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:20.636865 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:55:20.636999 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260619 20:55:20.645967 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:20.646595 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:20.649641 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: Generated new TSK 0
I20260619 20:55:20.649798 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:20.675171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:20.752862 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:20.753099 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:20.753139 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:20.754299 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:20.754364 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:20.754410 4541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:20.755831 4541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:20.756080 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:20.756276 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:20.761132 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:20.761236 4541 server_base.cc:1034] running on GCE node
W20260619 20:55:20.761165 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:20.761150 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:20.761610 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:20.761842 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:20.762979 4541 hybrid_clock.cc:648] HybridClock initialized: now 1781902520762966 us; error 28 us; skew 500 ppm
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:20.765280 4541 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:20.765645 4541 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:20.765698 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:20.765880 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:20.766969 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "7637ead6ceef4b9e8c66413a79481e66"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "8976799d6243661c99bc060064f8199d"
server_key_iv: "afcf2868d6812c055fa786107950e63c"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.767354 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7637ead6ceef4b9e8c66413a79481e66"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "8976799d6243661c99bc060064f8199d"
server_key_iv: "afcf2868d6812c055fa786107950e63c"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.768644 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:55:20.769372 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.769555 4541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:20.769652 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "7637ead6ceef4b9e8c66413a79481e66"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "8976799d6243661c99bc060064f8199d"
server_key_iv: "afcf2868d6812c055fa786107950e63c"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.769730 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:20.779134 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:20.780850 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:20.781009 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:20.781219 4541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:20.781518 4541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:20.781572 4541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.781611 4541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:20.781679 4541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.787428 4541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40531
I20260619 20:55:20.787460 4668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40531 every 8 connection(s)
I20260619 20:55:20.787734 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:20.789446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4541
I20260619 20:55:20.789563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:20.789776 392 external_mini_cluster.cc:1428] Setting key a35c53b748694c36b3962c2a4ed233b7
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:55:20.797080 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:20.788417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38605 (local address 127.0.98.62:44049)
0619 20:55:20.788490 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:20.788493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:20.788702 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:55:20.789141 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:20.789141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:20.789154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:20.789205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:20.790269 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.790867 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.791879 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.792110 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.792189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:20.794272 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:20.794273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:20.794273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:20.794281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:20.795702 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.796125 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.796126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.796127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.796185 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.796445 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.796445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.796446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.796557 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:20.796633 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:20.796897 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 20:55:20.796994 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:55:20.797297 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:20.788404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.1:38605)
0619 20:55:20.788579 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:20.788617 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:20.789063 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:20.789261 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:20.789268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:20.789505 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:20.790158 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:20.790165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.790972 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.790973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:20.791786 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:20.791788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.792199 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.792200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:20.792327 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:20.792817 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:20.792825 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:20.794134 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:20.795803 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:20.795810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:20.795811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:20.796034 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:20.796255 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:20.796256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:20.796256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:20.796369 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:20.796624 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:20.796630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:20.796768 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:55:20.796875 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:55:20.797027 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":114,"threads_started":1}
I20260619 20:55:20.797781 4669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260619 20:55:20.797887 4669 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:20.798038 4669 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260619 20:55:20.798594 4479 ts_manager.cc:194] Registered new tserver with Master: 7637ead6ceef4b9e8c66413a79481e66 (127.0.98.1:40531)
I20260619 20:55:20.799329 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:20.813498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:20.891716 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:20.891901 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:20.891932 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:20.893074 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:20.893114 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:20.893168 4674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:20.894631 4674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:20.894866 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:20.895092 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:20.899773 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:20.899927 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:20.899781 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:20.900158 4674 server_base.cc:1034] running on GCE node
I20260619 20:55:20.900326 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:20.900508 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:20.901670 4674 hybrid_clock.cc:648] HybridClock initialized: now 1781902520901671 us; error 53 us; skew 500 ppm
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:20.904214 4674 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:20.904541 4674 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:20.904621 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:20.904754 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:20.905776 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "9958d6212e1c443fa5ee02a706d3a230"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "9a24283e4fe625ac6bc70acd4cbe422f"
server_key_iv: "3cba50f944d6c794cdf90f1386d5d98b"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.906128 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9958d6212e1c443fa5ee02a706d3a230"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "9a24283e4fe625ac6bc70acd4cbe422f"
server_key_iv: "3cba50f944d6c794cdf90f1386d5d98b"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.907439 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:20.908129 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.908303 4674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:20.908370 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "9958d6212e1c443fa5ee02a706d3a230"
format_stamp: "Formatted at 2026-06-19 20:55:20 on dist-test-slave-8bgz"
server_key: "9a24283e4fe625ac6bc70acd4cbe422f"
server_key_iv: "3cba50f944d6c794cdf90f1386d5d98b"
server_key_version: "encryptionkey@0"
I20260619 20:55:20.908455 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:20.940604 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:20.942338 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:20.942503 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:20.942725 4674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:20.943037 4674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:20.943104 4674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.943137 4674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:20.943164 4674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:20.948745 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43923
I20260619 20:55:20.948798 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43923 every 8 connection(s)
I20260619 20:55:20.949041 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:20 dist-test-slave-8bgz krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:20.957402 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:20.949661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48435 (local address 127.0.98.62:44049)
0619 20:55:20.949717 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:20.949719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:20.949897 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:55:20.950434 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:20.950434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:20.950443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:20.950506 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:20.951353 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.951835 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.952761 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.952958 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.953019 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:20.954966 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:20.954967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:20.954968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:20.954973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:20.956320 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.956735 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.956736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.956736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.956785 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:20.956970 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:20.956970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:20.956971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:20.957055 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:20.957097 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:20.957288 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 20:55:20.957343 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:55:20.957604 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:20.949617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.2:48435)
0619 20:55:20.949766 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:20.949804 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:20.950349 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:20.950580 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:20.950584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:20.950820 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:20.951279 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:20.951284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.951910 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.951911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:20.952667 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:20.952669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:20.953031 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:20.953032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:20.953118 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:20.953575 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:20.953582 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:20.954891 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:20.956438 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:20.956444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:20.956446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:20.956675 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:20.956842 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:20.956843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:20.956843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:20.956932 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:20.957102 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:20.957105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:20.957206 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:55:20.957279 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:55:20.957383 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260619 20:55:20.958153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4674
I20260619 20:55:20.958271 4802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260619 20:55:20.958293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:20.958367 4802 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:20.958506 4802 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260619 20:55:20.958513 392 external_mini_cluster.cc:1428] Setting key b00e021465cc0f8641ed20e766946805
I20260619 20:55:20.958897 4479 ts_manager.cc:194] Registered new tserver with Master: 9958d6212e1c443fa5ee02a706d3a230 (127.0.98.2:43923)
I20260619 20:55:20.959439 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48435
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:20.981158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:21.060166 4807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:21.060359 4807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:21.060384 4807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:21.061534 4807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:21.061574 4807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:21.061600 4807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:21.063095 4807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:21.063334 4807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:21.063561 4807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:21.068325 4812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.068532 4807 server_base.cc:1034] running on GCE node
W20260619 20:55:21.068322 4813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.068325 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.068827 4807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:21.069069 4807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:21.070286 4807 hybrid_clock.cc:648] HybridClock initialized: now 1781902521070273 us; error 27 us; skew 500 ppm
Jun 19 20:55:21 dist-test-slave-8bgz krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:21.072784 4807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:21.073181 4807 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:21.073251 4807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:21.073387 4807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:21.074471 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "31946f8b39524e88b7d687dee7f3b666"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "6ebe282e853caf964884b31276ea337b"
server_key_iv: "22b7deb57155b99316485e787ab2fbbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.074824 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31946f8b39524e88b7d687dee7f3b666"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "6ebe282e853caf964884b31276ea337b"
server_key_iv: "22b7deb57155b99316485e787ab2fbbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.076088 4807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:21.076757 4821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.076926 4807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:21.077024 4807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "31946f8b39524e88b7d687dee7f3b666"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "6ebe282e853caf964884b31276ea337b"
server_key_iv: "22b7deb57155b99316485e787ab2fbbd"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.077105 4807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:21.087110 4807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:21.089298 4807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:21.089452 4807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:21.089659 4807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:21.089951 4807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:21.090013 4807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.090061 4807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:21.090096 4807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.095763 4807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34143
I20260619 20:55:21.095799 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34143 every 8 connection(s)
I20260619 20:55:21.096046 4807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:55:21 dist-test-slave-8bgz krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:21.104925 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.096701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60657 (local address 127.0.98.62:44049)
0619 20:55:21.096759 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.096761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.096991 (+ 230us) server_negotiation.cc:407] Connection header received
0619 20:55:21.097453 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.097454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.097469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.097520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.098410 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.098899 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.099847 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.100032 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.100081 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.102056 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.102057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.102058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:21.102064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.103602 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:21.103983 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:21.103984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:21.103985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:21.104046 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:21.104286 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:21.104287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:21.104287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:21.104401 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.104468 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.104754 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.104840 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:55:21.105063 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.096681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44049 (local address 127.0.98.3:60657)
0619 20:55:21.096849 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.096883 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.097374 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.097568 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.097571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.097802 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.098341 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.098346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.098969 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.098970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.099782 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.099784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.100098 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.100099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.100188 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.100591 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:21.100598 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.101916 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.103689 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:21.103695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:21.103697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:21.103906 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:21.104110 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:21.104110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:21.104111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:21.104218 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:21.104478 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.104484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.104612 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.104720 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.104861 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260619 20:55:21.105496 4935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44049
I20260619 20:55:21.105579 4935 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:21.105731 4935 heartbeater.cc:511] Master 127.0.98.62:44049 requested a full tablet report, sending...
I20260619 20:55:21.105731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4807
I20260619 20:55:21.105839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:21.106060 392 external_mini_cluster.cc:1428] Setting key 44940204af1685bc62ae99385cc01951
I20260619 20:55:21.106068 4479 ts_manager.cc:194] Registered new tserver with Master: 31946f8b39524e88b7d687dee7f3b666 (127.0.98.3:34143)
I20260619 20:55:21.106528 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60657
I20260619 20:55:21.106868 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:55:21.113375 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.108124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.0.98.62:44049)
0619 20:55:21.108181 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.108183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.108300 (+ 117us) server_negotiation.cc:407] Connection header received
0619 20:55:21.108335 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.108335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.108343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.108394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.108878 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.109309 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.109929 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.110119 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.110164 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.111260 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.111261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.111261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:21.111267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.112571 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:21.112827 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:21.112828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:21.112828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:21.112871 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:21.113001 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:21.113001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:21.113001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:21.113119 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.113183 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.113276 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.113324 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:21.114773 4479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:21.116201 4479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:21.122251 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40531 (local address 127.0.0.1:46462)
0619 20:55:21.118527 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.118536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.118579 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.118818 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.118819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.118824 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:21.118879 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.118881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.120040 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.120040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.121274 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.121275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.121944 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.121945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.122024 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.122039 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.122100 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.122148 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":61,"thread_start_us":70,"threads_started":1}
I20260619 20:55:21.122481 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.98.1:40531)
0619 20:55:21.118610 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.118614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.118622 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:21.118655 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.118657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.118670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.118737 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:21.118954 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.119953 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.121349 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.121866 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.121936 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.121988 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.122362 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.122405 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260619 20:55:21.122485 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34143 (local address 127.0.0.1:46860)
0619 20:55:21.118420 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.118461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.118529 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.118741 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.118743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.118745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:21.118802 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.118804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.119997 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.119997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.121233 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.121235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.122109 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.122110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.122171 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.122174 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.122259 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.122294 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":51,"thread_start_us":49,"threads_started":1}
I20260619 20:55:21.122776 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.0.98.3:34143)
0619 20:55:21.118334 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.118338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.118602 (+ 264us) server_negotiation.cc:407] Connection header received
0619 20:55:21.118638 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.118640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.118657 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.118738 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:21.118856 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.119905 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.121307 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.122028 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.122097 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.122148 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.122274 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.122705 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260619 20:55:21.123224 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43923 (local address 127.0.0.1:54494)
0619 20:55:21.118420 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.118462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.118529 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.118818 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.118819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.118821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:21.118880 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.118881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.120763 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.120764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.121752 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.121755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.122985 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.122986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.123063 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.123066 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.123122 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.123161 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":7,"thread_start_us":45,"threads_started":1}
I20260619 20:55:21.123137 4603 tablet_service.cc:1467] Processing CreateTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 (DEFAULT_TABLE table=test-table [id=67779280b65b4438aac68ff9cd93cd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.123481 4603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1156a3b8a37544f1a45ca198f6f9ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.123391 4869 tablet_service.cc:1467] Processing CreateTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 (DEFAULT_TABLE table=test-table [id=67779280b65b4438aac68ff9cd93cd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.123579 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.118190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.2:43923)
0619 20:55:21.118450 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.118455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.118602 (+ 147us) server_negotiation.cc:407] Connection header received
0619 20:55:21.118648 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.118650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.118665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.118734 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:21.120060 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.120670 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.121845 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.122830 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.122894 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.122953 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.123150 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.123501 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260619 20:55:21.123690 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1156a3b8a37544f1a45ca198f6f9ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.124195 4736 tablet_service.cc:1467] Processing CreateTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 (DEFAULT_TABLE table=test-table [id=67779280b65b4438aac68ff9cd93cd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.124476 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1156a3b8a37544f1a45ca198f6f9ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.126008 4949 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Bootstrap starting.
I20260619 20:55:21.126642 4949 tablet_bootstrap.cc:654] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.126837 4950 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Bootstrap starting.
I20260619 20:55:21.127264 4949 log.cc:824] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.127501 4950 tablet_bootstrap.cc:654] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.128144 4950 log.cc:824] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.128180 4949 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: No bootstrap required, opened a new log
I20260619 20:55:21.128232 4949 ts_tablet_manager.cc:1397] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:21.128917 4950 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: No bootstrap required, opened a new log
I20260619 20:55:21.128986 4950 ts_tablet_manager.cc:1397] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:21.130352 4952 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Bootstrap starting.
I20260619 20:55:21.130302 4949 raft_consensus.cc:348] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.130463 4949 raft_consensus.cc:374] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.130505 4949 raft_consensus.cc:729] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7637ead6ceef4b9e8c66413a79481e66, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.130615 4949 consensus_queue.cc:260] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.130893 4949 ts_tablet_manager.cc:1428] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:21.130992 4952 tablet_bootstrap.cc:654] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.131028 4669 heartbeater.cc:503] Master 127.0.98.62:44049 was elected leader, sending a full tablet report...
I20260619 20:55:21.131094 4950 raft_consensus.cc:348] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.131239 4950 raft_consensus.cc:374] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.131397 4950 raft_consensus.cc:729] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31946f8b39524e88b7d687dee7f3b666, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.131508 4950 consensus_queue.cc:260] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.131592 4952 log.cc:824] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.131768 4950 ts_tablet_manager.cc:1428] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:21.131889 4935 heartbeater.cc:503] Master 127.0.98.62:44049 was elected leader, sending a full tablet report...
I20260619 20:55:21.132323 4952 tablet_bootstrap.cc:492] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: No bootstrap required, opened a new log
I20260619 20:55:21.132395 4952 ts_tablet_manager.cc:1397] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:21.133735 4952 raft_consensus.cc:348] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.133905 4952 raft_consensus.cc:374] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.133965 4952 raft_consensus.cc:729] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9958d6212e1c443fa5ee02a706d3a230, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.134074 4952 consensus_queue.cc:260] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.134346 4952 ts_tablet_manager.cc:1428] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:55:21.134367 4802 heartbeater.cc:503] Master 127.0.98.62:44049 was elected leader, sending a full tablet report...
W20260619 20:55:21.199525 4803 tablet.cc:2369] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:21.288566 4670 tablet.cc:2369] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:21.302256 4955 raft_consensus.cc:482] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:21.302366 4955 raft_consensus.cc:504] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.302695 4955 leader_election.cc:288] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9958d6212e1c443fa5ee02a706d3a230 (127.0.98.2:43923), 7637ead6ceef4b9e8c66413a79481e66 (127.0.98.1:40531)
I20260619 20:55:21.305590 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.302937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48937 (local address 127.0.98.2:43923)
0619 20:55:21.302999 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.303001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.303009 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:21.303082 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.303083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.303095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.303145 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:21.303305 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.303889 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.304836 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.305224 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.305301 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.305311 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.305479 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.305515 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:55:21.305615 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.303053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56189 (local address 127.0.98.1:40531)
0619 20:55:21.303100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.303102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.303211 (+ 109us) server_negotiation.cc:407] Connection header received
0619 20:55:21.303300 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.303302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.303312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.303380 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:21.303557 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.304002 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.304917 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.305271 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.305351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.305360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.305514 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.305550 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:21.305629 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.302893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43923 (local address 127.0.98.3:48937)
0619 20:55:21.302931 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.302942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.303025 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.303170 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.303171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.303173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:21.303246 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.303248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.303966 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.303967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.304750 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.304752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.305310 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.305311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.305392 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.305395 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.305471 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.305515 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":28}
I20260619 20:55:21.305810 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.303004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40531 (local address 127.0.98.3:56189)
0619 20:55:21.303144 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.303170 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.303221 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.303395 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.303396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.303397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:21.303489 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.303491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.304090 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.304091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.304832 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:21.304834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.305351 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.305352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:21.305425 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.305428 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.305506 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.305542 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260619 20:55:21.306279 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1156a3b8a37544f1a45ca198f6f9ae67" candidate_uuid: "31946f8b39524e88b7d687dee7f3b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7637ead6ceef4b9e8c66413a79481e66" is_pre_election: true
I20260619 20:55:21.306294 4756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1156a3b8a37544f1a45ca198f6f9ae67" candidate_uuid: "31946f8b39524e88b7d687dee7f3b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9958d6212e1c443fa5ee02a706d3a230" is_pre_election: true
I20260619 20:55:21.306432 4623 raft_consensus.cc:2454] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31946f8b39524e88b7d687dee7f3b666 in term 0.
I20260619 20:55:21.306442 4756 raft_consensus.cc:2454] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31946f8b39524e88b7d687dee7f3b666 in term 0.
I20260619 20:55:21.306623 4825 leader_election.cc:302] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31946f8b39524e88b7d687dee7f3b666, 9958d6212e1c443fa5ee02a706d3a230; no voters:
I20260619 20:55:21.306761 4955 raft_consensus.cc:2781] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:21.306810 4955 raft_consensus.cc:482] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:21.306826 4955 raft_consensus.cc:3037] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:21.307540 4955 raft_consensus.cc:504] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.307660 4955 leader_election.cc:288] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [CANDIDATE]: Term 1 election: Requested vote from peers 9958d6212e1c443fa5ee02a706d3a230 (127.0.98.2:43923), 7637ead6ceef4b9e8c66413a79481e66 (127.0.98.1:40531)
I20260619 20:55:21.307793 4756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1156a3b8a37544f1a45ca198f6f9ae67" candidate_uuid: "31946f8b39524e88b7d687dee7f3b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9958d6212e1c443fa5ee02a706d3a230"
I20260619 20:55:21.307829 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1156a3b8a37544f1a45ca198f6f9ae67" candidate_uuid: "31946f8b39524e88b7d687dee7f3b666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7637ead6ceef4b9e8c66413a79481e66"
I20260619 20:55:21.307876 4756 raft_consensus.cc:3037] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:21.307893 4623 raft_consensus.cc:3037] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:21.308753 4756 raft_consensus.cc:2454] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31946f8b39524e88b7d687dee7f3b666 in term 1.
I20260619 20:55:21.308753 4623 raft_consensus.cc:2454] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31946f8b39524e88b7d687dee7f3b666 in term 1.
I20260619 20:55:21.308912 4825 leader_election.cc:302] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31946f8b39524e88b7d687dee7f3b666, 9958d6212e1c443fa5ee02a706d3a230; no voters:
I20260619 20:55:21.309015 4955 raft_consensus.cc:2781] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:21.309154 4955 raft_consensus.cc:686] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 1 LEADER]: Becoming Leader. State: Replica: 31946f8b39524e88b7d687dee7f3b666, State: Running, Role: LEADER
I20260619 20:55:21.309245 4955 consensus_queue.cc:237] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } }
I20260619 20:55:21.310070 4478 catalog_manager.cc:5495] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31946f8b39524e88b7d687dee7f3b666 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "31946f8b39524e88b7d687dee7f3b666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31946f8b39524e88b7d687dee7f3b666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 } health_report { overall_health: UNKNOWN } } }
W20260619 20:55:21.346545 4936 tablet.cc:2369] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:21.346930 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.344662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.0.98.3:34143)
0619 20:55:21.344729 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.344731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.344738 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:21.344767 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.344767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.344784 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.344842 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:21.344935 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.345395 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.346074 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:21.346308 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:21.346376 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:21.346535 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:21.346645 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:21.346724 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.346801 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.346818 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:55:21.348556 4623 raft_consensus.cc:1261] T 1156a3b8a37544f1a45ca198f6f9ae67 P 7637ead6ceef4b9e8c66413a79481e66 [term 1 FOLLOWER]: Refusing update from remote peer 31946f8b39524e88b7d687dee7f3b666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:21.348556 4756 raft_consensus.cc:1261] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 1 FOLLOWER]: Refusing update from remote peer 31946f8b39524e88b7d687dee7f3b666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:21.348939 4959 consensus_queue.cc:1035] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7637ead6ceef4b9e8c66413a79481e66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:21.349066 4955 consensus_queue.cc:1035] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9958d6212e1c443fa5ee02a706d3a230" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:21.351827 4962 mvcc.cc:205] Tried to move back new op lower bound from 7298672727441051648 to 7298672727283253248. Current Snapshot: MvccSnapshot[applied={T|T < 7298672727441051648 or (T in {7298672727441051648})}]
I20260619 20:55:21.353255 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:21.353331 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:21.354080 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 495e28a2553e45188aad1b24f99406f6: Sending DeleteTablet for 3 replicas of tablet 1156a3b8a37544f1a45ca198f6f9ae67
I20260619 20:55:21.354483 4869 tablet_service.cc:1514] Processing DeleteTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46860
I20260619 20:55:21.354492 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54494
I20260619 20:55:21.354694 4973 tablet_replica.cc:330] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: stopping tablet replica
I20260619 20:55:21.354691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4541
I20260619 20:55:21.354735 4603 tablet_service.cc:1514] Processing DeleteTablet for tablet 1156a3b8a37544f1a45ca198f6f9ae67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260619 20:55:21.354835 4973 raft_consensus.cc:2229] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:21.354931 4973 raft_consensus.cc:2258] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:21.355060 4972 tablet_replica.cc:330] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: stopping tablet replica
I20260619 20:55:21.355185 4972 raft_consensus.cc:2229] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:21.355296 4972 raft_consensus.cc:2258] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:21.355628 4973 ts_tablet_manager.cc:1905] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:21.355715 4972 ts_tablet_manager.cc:1905] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:21.357442 4972 ts_tablet_manager.cc:1918] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:21.357542 4973 ts_tablet_manager.cc:1918] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:21.357635 4973 log.cc:1196] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/1156a3b8a37544f1a45ca198f6f9ae67
I20260619 20:55:21.357635 4972 log.cc:1196] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/1156a3b8a37544f1a45ca198f6f9ae67
I20260619 20:55:21.357928 4973 ts_tablet_manager.cc:1939] T 1156a3b8a37544f1a45ca198f6f9ae67 P 31946f8b39524e88b7d687dee7f3b666: Deleting consensus metadata
I20260619 20:55:21.358011 4972 ts_tablet_manager.cc:1939] T 1156a3b8a37544f1a45ca198f6f9ae67 P 9958d6212e1c443fa5ee02a706d3a230: Deleting consensus metadata
I20260619 20:55:21.358335 4463 catalog_manager.cc:4841] TS 31946f8b39524e88b7d687dee7f3b666 (127.0.98.3:34143): tablet 1156a3b8a37544f1a45ca198f6f9ae67 (table test-table [id=67779280b65b4438aac68ff9cd93cd50]) successfully deleted
I20260619 20:55:21.358392 4466 catalog_manager.cc:4841] TS 9958d6212e1c443fa5ee02a706d3a230 (127.0.98.2:43923): tablet 1156a3b8a37544f1a45ca198f6f9ae67 (table test-table [id=67779280b65b4438aac68ff9cd93cd50]) successfully deleted
W20260619 20:55:21.360688 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40531 (error 108)
I20260619 20:55:21.360901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4674
I20260619 20:55:21.361492 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.360973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40531 (local address 127.0.0.1:46478)
0619 20:55:21.361419 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.361453 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260619 20:55:21.362084 4466 catalog_manager.cc:4584] TS 7637ead6ceef4b9e8c66413a79481e66 (127.0.98.1:40531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1156a3b8a37544f1a45ca198f6f9ae67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40531: connect: Connection refused (error 111)
I20260619 20:55:21.366453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4807
I20260619 20:55:21.371179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4448
2026-06-19T20:55:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2511 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T20:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:21Z Disabled control of system clock
I20260619 20:55:21.397385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40977
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:55:21.476593 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:21.476774 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:21.476794 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:21.477813 4978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:21.477843 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:21.477856 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:21.477869 4978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:21.477880 4978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:21.479184 4978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:21.479373 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:21.479593 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:21.484344 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.484404 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.484344 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.484681 4978 server_base.cc:1034] running on GCE node
I20260619 20:55:21.484927 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:21.485213 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:21.486351 4978 hybrid_clock.cc:648] HybridClock initialized: now 1781902521486338 us; error 27 us; skew 500 ppm
I20260619 20:55:21.487588 4978 webserver.cc:460] Webserver started at http://127.0.98.62:40485/ using document root <none> and password file <none>
I20260619 20:55:21.487834 4978 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:21.487881 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:21.488013 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:21.489038 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "fe81b4d5430e4fdab4e820e7ef189969"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "a300cf8b1cfdadc2a32678cc893b3605"
server_key_iv: "f883aeb1843bbfbd14b579707b342428"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.489387 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe81b4d5430e4fdab4e820e7ef189969"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "a300cf8b1cfdadc2a32678cc893b3605"
server_key_iv: "f883aeb1843bbfbd14b579707b342428"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.490695 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:21.491370 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.491545 4978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:21.491612 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "fe81b4d5430e4fdab4e820e7ef189969"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "a300cf8b1cfdadc2a32678cc893b3605"
server_key_iv: "f883aeb1843bbfbd14b579707b342428"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.491662 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:21.494781 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:21.494980 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:21.495070 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:21.498615 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40977
I20260619 20:55:21.498682 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40977 every 8 connection(s)
I20260619 20:55:21.498965 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:21.499572 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.501004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 4978
I20260619 20:55:21.501083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:21.501262 392 external_mini_cluster.cc:1428] Setting key 892ae5a136d787e8890c52e6a3111c2f
I20260619 20:55:21.502578 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: Bootstrap starting.
I20260619 20:55:21.503229 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.503523 5045 log.cc:824] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.503882 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.501783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.62:40977)
0619 20:55:21.501979 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.501987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.502019 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:55:21.502549 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.502558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.502610 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.502755 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.503371 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.503375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.503375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.503379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.503429 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.503439 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.503534 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.503625 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.503685 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260619 20:55:21.504272 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: No bootstrap required, opened a new log
I20260619 20:55:21.505596 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 20:55:21.505815 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.505863 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe81b4d5430e4fdab4e820e7ef189969, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.505991 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 20:55:21.506074 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:21.506112 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:21.506158 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:21.506883 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 20:55:21.506991 5045 leader_election.cc:302] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe81b4d5430e4fdab4e820e7ef189969; no voters:
I20260619 20:55:21.507128 5045 leader_election.cc:288] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:21.507223 5050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:21.507372 5050 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [term 1 LEADER]: Becoming Leader. State: Replica: fe81b4d5430e4fdab4e820e7ef189969, State: Running, Role: LEADER
I20260619 20:55:21.507414 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:21.507460 5050 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 20:55:21.507809 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe81b4d5430e4fdab4e820e7ef189969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } } }
I20260619 20:55:21.507952 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:21.507840 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe81b4d5430e4fdab4e820e7ef189969. Latest consensus state: current_term: 1 leader_uuid: "fe81b4d5430e4fdab4e820e7ef189969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81b4d5430e4fdab4e820e7ef189969" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } } }
I20260619 20:55:21.508245 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:21.508337 5059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:21.508796 5059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:21.510345 5059 catalog_manager.cc:1347] Generated new cluster ID: 97ac27a8381a438daddce4f0a283f0d8
I20260619 20:55:21.510399 5059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:21.541200 5059 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:21.541383 5059 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:21.548810 5059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: Generated new TSK 0
I20260619 20:55:21.548976 5059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:21.554385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:55:21.632623 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:21.632831 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:21.632869 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:21.633999 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:21.634058 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:21.634101 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:21.635456 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:21.635715 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:21.635993 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:21.640781 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.640797 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.640793 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.640985 5069 server_base.cc:1034] running on GCE node
I20260619 20:55:21.641206 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:21.641382 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:21.642515 5069 hybrid_clock.cc:648] HybridClock initialized: now 1781902521642493 us; error 36 us; skew 500 ppm
I20260619 20:55:21.643743 5069 webserver.cc:460] Webserver started at http://127.0.98.1:37617/ using document root <none> and password file <none>
I20260619 20:55:21.643985 5069 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:21.644032 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:21.644205 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:21.645191 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "90a9272d1fb547e29b65e4a6235dac3c"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0a2488c050c2c6cfc1aa95563c839306"
server_key_iv: "f32044923f23dd01d8ac784cddd13d7d"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.645522 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "90a9272d1fb547e29b65e4a6235dac3c"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0a2488c050c2c6cfc1aa95563c839306"
server_key_iv: "f32044923f23dd01d8ac784cddd13d7d"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.646848 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:21.647521 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.647672 5069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:21.647765 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "90a9272d1fb547e29b65e4a6235dac3c"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0a2488c050c2c6cfc1aa95563c839306"
server_key_iv: "f32044923f23dd01d8ac784cddd13d7d"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.647837 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:21.651021 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:21.651271 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:21.651413 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:21.651628 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:21.651933 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:21.651985 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.652026 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:21.652053 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.657426 5069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43601
I20260619 20:55:21.657490 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43601 every 8 connection(s)
I20260619 20:55:21.657801 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:21.658833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5069
I20260619 20:55:21.658936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:21.659111 392 external_mini_cluster.cc:1428] Setting key 200ea2ea7ae8ece5eb80bf7c16a9b92c
I20260619 20:55:21.660064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 20:55:21.660959 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.658494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38093 (local address 127.0.98.62:40977)
0619 20:55:21.658541 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.658543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.658759 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:55:21.659397 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.659399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.659405 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.659465 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.660729 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.660730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.660730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.660734 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.660749 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.660753 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.660802 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.660900 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.660910 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:21.661195 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.658445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40977 (local address 127.0.98.1:38093)
0619 20:55:21.658625 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.658666 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.659275 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.659533 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.659536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.659546 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.660568 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.660569 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.660605 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.660637 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.660821 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.660826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.660828 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.660907 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.660965 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260619 20:55:21.661710 5197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40977
I20260619 20:55:21.661808 5197 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:21.661947 5197 heartbeater.cc:511] Master 127.0.98.62:40977 requested a full tablet report, sending...
I20260619 20:55:21.662518 5009 ts_manager.cc:194] Registered new tserver with Master: 90a9272d1fb547e29b65e4a6235dac3c (127.0.98.1:43601)
W20260619 20:55:21.738809 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:21.738988 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:21.739009 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:21.740027 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:21.740083 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:21.740108 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:21.741324 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:21.741483 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:21.741725 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:21.746457 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.746457 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.746557 5200 server_base.cc:1034] running on GCE node
W20260619 20:55:21.746483 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.746896 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:21.747138 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:21.748299 5200 hybrid_clock.cc:648] HybridClock initialized: now 1781902521748287 us; error 45 us; skew 500 ppm
I20260619 20:55:21.749523 5200 webserver.cc:460] Webserver started at http://127.0.98.2:36349/ using document root <none> and password file <none>
I20260619 20:55:21.749765 5200 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:21.749836 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:21.749970 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:21.751088 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "4aa027a05a704601aed873680d11ed97"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0ce7e26419838350c02fc3ebf32a522f"
server_key_iv: "010189721a552f06168f92035ae10c37"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.751474 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa027a05a704601aed873680d11ed97"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0ce7e26419838350c02fc3ebf32a522f"
server_key_iv: "010189721a552f06168f92035ae10c37"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.752696 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:21.753371 5214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.753535 5200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:21.753631 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "4aa027a05a704601aed873680d11ed97"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "0ce7e26419838350c02fc3ebf32a522f"
server_key_iv: "010189721a552f06168f92035ae10c37"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.753717 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:21.756958 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:21.757217 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:21.757360 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:21.757573 5200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:21.757871 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:21.757925 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.757963 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:21.757992 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.763752 5200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35899
I20260619 20:55:21.763788 5327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35899 every 8 connection(s)
I20260619 20:55:21.764111 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:21.764863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5200
I20260619 20:55:21.764982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:21.765202 392 external_mini_cluster.cc:1428] Setting key 26cdc84e33a9a97aea05e9c1d9007805
I20260619 20:55:21.766150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 20:55:21.767637 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.764818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58983 (local address 127.0.98.62:40977)
0619 20:55:21.764888 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.764890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.765191 (+ 301us) server_negotiation.cc:407] Connection header received
0619 20:55:21.765852 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.765853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.765857 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.765905 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.767357 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.767358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.767358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.767362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.767376 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.767379 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.767457 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.767550 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.767567 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:21.767833 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.764802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40977 (local address 127.0.98.2:58983)
0619 20:55:21.765038 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.765083 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.765668 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.766002 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.766006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.766027 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.767197 (+ 1170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.767197 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.767234 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.767269 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.767457 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.767464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.767467 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.767552 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.767615 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260619 20:55:21.768436 5328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40977
I20260619 20:55:21.768537 5328 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:21.768675 5328 heartbeater.cc:511] Master 127.0.98.62:40977 requested a full tablet report, sending...
I20260619 20:55:21.769011 5009 ts_manager.cc:194] Registered new tserver with Master: 4aa027a05a704601aed873680d11ed97 (127.0.98.2:35899)
W20260619 20:55:21.845141 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:21.845350 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:21.845382 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:21.846521 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:21.846588 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:21.846632 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:21.847939 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:21.848189 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:21.848456 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:21.853338 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.853325 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:21.853314 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:21.853433 5331 server_base.cc:1034] running on GCE node
I20260619 20:55:21.853721 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:21.853904 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:21.855031 5331 hybrid_clock.cc:648] HybridClock initialized: now 1781902521855015 us; error 28 us; skew 500 ppm
I20260619 20:55:21.856341 5331 webserver.cc:460] Webserver started at http://127.0.98.3:44651/ using document root <none> and password file <none>
I20260619 20:55:21.856585 5331 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:21.856630 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:21.856717 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:21.857790 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "59ab1a05b2f040e5a74c07287b3cb874"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "64d8614d8f66c8c791b70aea529dc207"
server_key_iv: "866b3ba3e3fdc1eebd4ed9796bf77e33"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.858090 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59ab1a05b2f040e5a74c07287b3cb874"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "64d8614d8f66c8c791b70aea529dc207"
server_key_iv: "866b3ba3e3fdc1eebd4ed9796bf77e33"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.859426 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:21.860066 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.860227 5331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:21.860324 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "59ab1a05b2f040e5a74c07287b3cb874"
format_stamp: "Formatted at 2026-06-19 20:55:21 on dist-test-slave-8bgz"
server_key: "64d8614d8f66c8c791b70aea529dc207"
server_key_iv: "866b3ba3e3fdc1eebd4ed9796bf77e33"
server_key_version: "encryptionkey@0"
I20260619 20:55:21.860401 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:21.863718 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:21.863957 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:21.864101 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:21.864305 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:21.864595 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:21.864648 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.864687 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:21.864715 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:21.870182 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42111
I20260619 20:55:21.870256 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42111 every 8 connection(s)
I20260619 20:55:21.870572 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:21.871306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5331
I20260619 20:55:21.871407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:21.871587 392 external_mini_cluster.cc:1428] Setting key 4ef24b67a54ce2edbb9d20c078b7e82d
I20260619 20:55:21.873553 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.871220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48397 (local address 127.0.98.62:40977)
0619 20:55:21.871259 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.871261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.871527 (+ 266us) server_negotiation.cc:407] Connection header received
0619 20:55:21.872102 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.872103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.872109 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.872156 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.873327 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.873328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.873328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.873333 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.873348 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.873352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.873397 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.873468 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.873482 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:21.873742 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.871223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40977 (local address 127.0.98.3:48397)
0619 20:55:21.871411 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.871439 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.872000 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.872231 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.872235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.872244 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.873180 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.873180 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.873206 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.873232 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.873406 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.873411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.873414 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.873480 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.873537 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260619 20:55:21.874112 5459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40977
I20260619 20:55:21.874207 5459 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:21.874365 5459 heartbeater.cc:511] Master 127.0.98.62:40977 requested a full tablet report, sending...
I20260619 20:55:21.874683 5009 ts_manager.cc:194] Registered new tserver with Master: 59ab1a05b2f040e5a74c07287b3cb874 (127.0.98.3:42111)
I20260619 20:55:21.883086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:55:21.885236 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.884269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.0.98.62:40977)
0619 20:55:21.884326 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.884328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.884449 (+ 121us) server_negotiation.cc:407] Connection header received
0619 20:55:21.884490 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.884491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.884494 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.884541 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.885016 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.885017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.885017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.885021 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.885032 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.885035 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.885086 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.885124 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.885135 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:55:21.886889 5009 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:21.888334 5009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:21.891997 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.0.98.2:35899)
0619 20:55:21.890645 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.890649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.890794 (+ 145us) server_negotiation.cc:407] Connection header received
0619 20:55:21.890832 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.890834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.890847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.890950 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.891628 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.891635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.891635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.891639 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.891674 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.891700 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.891738 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.891890 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.891909 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260619 20:55:21.892179 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35899 (local address 127.0.0.1:58804)
0619 20:55:21.890705 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.890715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.890774 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.890965 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.890971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.890977 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.891485 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.891485 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.891503 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.891527 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.891786 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.891793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.891796 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.892105 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.892122 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":85,"threads_started":1}
I20260619 20:55:21.892334 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.0.1:34726)
0619 20:55:21.890476 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.890515 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.890779 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.891477 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.891477 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.891480 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.891878 (+ 398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.891878 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.891885 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.891897 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.892192 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.892192 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.892193 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.892254 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.892268 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260619 20:55:21.892522 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.0.98.3:42111)
0619 20:55:21.890890 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.890894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.890910 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:55:21.890930 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.890931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.890938 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.891029 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.891986 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.891993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.891994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.891998 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.892030 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.892056 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.892196 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.892262 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.892276 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":80,"threads_started":1}
I20260619 20:55:21.893008 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43601 (local address 127.0.0.1:60182)
0619 20:55:21.891339 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:21.891349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:21.891399 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:21.891802 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:21.891803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:21.891806 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:21.892515 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:21.892516 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:21.892522 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:21.892533 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:21.892882 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:21.892883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:21.892884 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:21.892939 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:55:21.892951 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":66,"threads_started":1}
I20260619 20:55:21.892944 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd (DEFAULT_TABLE table=test-table [id=993be8bca46a4662b676bf29a2759049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.893205 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:21.890618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.0.98.1:43601)
0619 20:55:21.890831 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:21.890835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:21.891396 (+ 561us) server_negotiation.cc:407] Connection header received
0619 20:55:21.891458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:21.891460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:21.891469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:21.891547 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:21.892719 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:21.892725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:21.892725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:21.892728 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:21.892779 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:21.892803 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:21.892866 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:21.892951 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:55:21.892966 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260619 20:55:21.893172 5393 tablet_service.cc:1467] Processing CreateTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd (DEFAULT_TABLE table=test-table [id=993be8bca46a4662b676bf29a2759049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.893347 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b865087e6438eae3b0b7ed8f66dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.893527 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b865087e6438eae3b0b7ed8f66dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.893812 5131 tablet_service.cc:1467] Processing CreateTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd (DEFAULT_TABLE table=test-table [id=993be8bca46a4662b676bf29a2759049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:21.894116 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b865087e6438eae3b0b7ed8f66dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:21.896203 5473 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Bootstrap starting.
I20260619 20:55:21.896678 5474 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Bootstrap starting.
I20260619 20:55:21.896854 5473 tablet_bootstrap.cc:654] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.897153 5475 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Bootstrap starting.
I20260619 20:55:21.897179 5473 log.cc:824] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.897312 5474 tablet_bootstrap.cc:654] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.897634 5474 log.cc:824] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.897773 5475 tablet_bootstrap.cc:654] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:21.898061 5475 log.cc:824] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:21.898357 5474 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: No bootstrap required, opened a new log
I20260619 20:55:21.898433 5474 ts_tablet_manager.cc:1397] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:21.898829 5473 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: No bootstrap required, opened a new log
I20260619 20:55:21.898898 5473 ts_tablet_manager.cc:1397] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:55:21.901090 5474 raft_consensus.cc:348] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.901270 5474 raft_consensus.cc:374] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.901304 5474 raft_consensus.cc:729] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ab1a05b2f040e5a74c07287b3cb874, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.901422 5474 consensus_queue.cc:260] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.901424 5473 raft_consensus.cc:348] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.901584 5473 raft_consensus.cc:374] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.901631 5473 raft_consensus.cc:729] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa027a05a704601aed873680d11ed97, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.901678 5474 ts_tablet_manager.cc:1428] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:21.901737 5473 consensus_queue.cc:260] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.901811 5459 heartbeater.cc:503] Master 127.0.98.62:40977 was elected leader, sending a full tablet report...
I20260619 20:55:21.902001 5328 heartbeater.cc:503] Master 127.0.98.62:40977 was elected leader, sending a full tablet report...
I20260619 20:55:21.901986 5473 ts_tablet_manager.cc:1428] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:21.902796 5475 tablet_bootstrap.cc:492] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: No bootstrap required, opened a new log
I20260619 20:55:21.902863 5475 ts_tablet_manager.cc:1397] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 20:55:21.904601 5475 raft_consensus.cc:348] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.904739 5475 raft_consensus.cc:374] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:21.904783 5475 raft_consensus.cc:729] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a9272d1fb547e29b65e4a6235dac3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:21.904896 5475 consensus_queue.cc:260] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:21.905129 5475 ts_tablet_manager.cc:1428] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:55:21.905179 5197 heartbeater.cc:503] Master 127.0.98.62:40977 was elected leader, sending a full tablet report...
W20260619 20:55:21.908465 5198 tablet.cc:2369] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:22.014653 5329 tablet.cc:2369] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:22.121140 5460 tablet.cc:2369] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:22.133795 5481 raft_consensus.cc:482] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:22.133921 5481 raft_consensus.cc:504] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:22.134259 5481 leader_election.cc:288] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59ab1a05b2f040e5a74c07287b3cb874 (127.0.98.3:42111), 4aa027a05a704601aed873680d11ed97 (127.0.98.2:35899)
I20260619 20:55:22.135696 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.134584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40333 (local address 127.0.98.3:42111)
0619 20:55:22.134642 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.134645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.134798 (+ 153us) server_negotiation.cc:407] Connection header received
0619 20:55:22.134825 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.134826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.134831 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.134884 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.135464 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.135465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.135466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.135470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.135485 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.135490 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.135547 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.135611 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.135626 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:22.135699 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.134575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38311 (local address 127.0.98.2:35899)
0619 20:55:22.134622 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.134625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.134636 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:22.134667 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.134667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.134673 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.134748 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.135405 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.135407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.135407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.135428 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.135451 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.135456 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.135521 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.135626 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.135639 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:55:22.135717 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.134488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35899 (local address 127.0.98.1:38311)
0619 20:55:22.134536 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.134548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:22.134588 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:22.134777 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:22.134778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:22.134780 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:22.135261 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:22.135261 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:22.135268 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:22.135298 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:22.135540 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:22.135541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:22.135542 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:22.135599 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:55:22.135619 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260619 20:55:22.135728 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.134488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42111 (local address 127.0.98.1:40333)
0619 20:55:22.134698 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.134710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:22.134772 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:22.134908 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:22.134909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:22.134911 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:22.135366 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:22.135367 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:22.135372 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:22.135381 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:22.135552 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:22.135553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:22.135554 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:22.135603 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:55:22.135617 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":56,"thread_start_us":112,"threads_started":1}
I20260619 20:55:22.136087 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b865087e6438eae3b0b7ed8f66dbd" candidate_uuid: "90a9272d1fb547e29b65e4a6235dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ab1a05b2f040e5a74c07287b3cb874" is_pre_election: true
I20260619 20:55:22.136257 5413 raft_consensus.cc:2454] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a9272d1fb547e29b65e4a6235dac3c in term 0.
I20260619 20:55:22.136279 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b865087e6438eae3b0b7ed8f66dbd" candidate_uuid: "90a9272d1fb547e29b65e4a6235dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa027a05a704601aed873680d11ed97" is_pre_election: true
I20260619 20:55:22.136415 5282 raft_consensus.cc:2454] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a9272d1fb547e29b65e4a6235dac3c in term 0.
I20260619 20:55:22.136500 5085 leader_election.cc:302] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ab1a05b2f040e5a74c07287b3cb874, 90a9272d1fb547e29b65e4a6235dac3c; no voters:
I20260619 20:55:22.136636 5481 raft_consensus.cc:2781] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:22.136730 5481 raft_consensus.cc:482] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:22.136766 5481 raft_consensus.cc:3037] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.137473 5481 raft_consensus.cc:504] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:22.137627 5481 leader_election.cc:288] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [CANDIDATE]: Term 1 election: Requested vote from peers 59ab1a05b2f040e5a74c07287b3cb874 (127.0.98.3:42111), 4aa027a05a704601aed873680d11ed97 (127.0.98.2:35899)
I20260619 20:55:22.137768 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b865087e6438eae3b0b7ed8f66dbd" candidate_uuid: "90a9272d1fb547e29b65e4a6235dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa027a05a704601aed873680d11ed97"
I20260619 20:55:22.137801 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b865087e6438eae3b0b7ed8f66dbd" candidate_uuid: "90a9272d1fb547e29b65e4a6235dac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ab1a05b2f040e5a74c07287b3cb874"
I20260619 20:55:22.137847 5282 raft_consensus.cc:3037] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.137897 5413 raft_consensus.cc:3037] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.138638 5282 raft_consensus.cc:2454] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a9272d1fb547e29b65e4a6235dac3c in term 1.
I20260619 20:55:22.138736 5413 raft_consensus.cc:2454] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a9272d1fb547e29b65e4a6235dac3c in term 1.
I20260619 20:55:22.138808 5087 leader_election.cc:302] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa027a05a704601aed873680d11ed97, 90a9272d1fb547e29b65e4a6235dac3c; no voters:
I20260619 20:55:22.138895 5481 raft_consensus.cc:2781] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:22.139063 5481 raft_consensus.cc:686] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [term 1 LEADER]: Becoming Leader. State: Replica: 90a9272d1fb547e29b65e4a6235dac3c, State: Running, Role: LEADER
I20260619 20:55:22.139156 5481 consensus_queue.cc:237] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } }
I20260619 20:55:22.139930 5008 catalog_manager.cc:5495] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a9272d1fb547e29b65e4a6235dac3c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "90a9272d1fb547e29b65e4a6235dac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a9272d1fb547e29b65e4a6235dac3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43601 } health_report { overall_health: HEALTHY } } }
I20260619 20:55:22.175056 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.174089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.98.1:43601)
0619 20:55:22.174162 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.174164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.174175 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:22.174240 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.174241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.174261 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.174327 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.174835 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.174837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.174837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.174842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.174856 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.174859 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.174913 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.174971 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.174983 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:55:22.176676 5413 raft_consensus.cc:1261] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 1 FOLLOWER]: Refusing update from remote peer 90a9272d1fb547e29b65e4a6235dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:22.176684 5282 raft_consensus.cc:1261] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 1 FOLLOWER]: Refusing update from remote peer 90a9272d1fb547e29b65e4a6235dac3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:22.177044 5483 consensus_queue.cc:1035] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa027a05a704601aed873680d11ed97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:22.177181 5481 consensus_queue.cc:1035] T 153b865087e6438eae3b0b7ed8f66dbd P 90a9272d1fb547e29b65e4a6235dac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ab1a05b2f040e5a74c07287b3cb874" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:22.179421 5486 mvcc.cc:205] Tried to move back new op lower bound from 7298672730833162240 to 7298672730682163200. Current Snapshot: MvccSnapshot[applied={T|T < 7298672730833162240}]
I20260619 20:55:22.179636 5488 mvcc.cc:205] Tried to move back new op lower bound from 7298672730833162240 to 7298672730682163200. Current Snapshot: MvccSnapshot[applied={T|T < 7298672730833162240}]
I20260619 20:55:22.181845 5008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:22.181937 5008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:22.182610 5008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe81b4d5430e4fdab4e820e7ef189969: Sending DeleteTablet for 3 replicas of tablet 153b865087e6438eae3b0b7ed8f66dbd
I20260619 20:55:22.182951 5393 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:22 UTC) from {username='slave'} at 127.0.0.1:34726
I20260619 20:55:22.182951 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:22 UTC) from {username='slave'} at 127.0.0.1:60182
I20260619 20:55:22.183126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5069
I20260619 20:55:22.183192 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b865087e6438eae3b0b7ed8f66dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:22 UTC) from {username='slave'} at 127.0.0.1:58804
I20260619 20:55:22.183768 5498 tablet_replica.cc:330] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: stopping tablet replica
I20260619 20:55:22.183902 5498 raft_consensus.cc:2229] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:22.184037 5498 raft_consensus.cc:2258] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:22.184432 5498 ts_tablet_manager.cc:1905] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:22.186108 5498 ts_tablet_manager.cc:1918] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:22.186185 5498 log.cc:1196] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/153b865087e6438eae3b0b7ed8f66dbd
I20260619 20:55:22.186425 5498 ts_tablet_manager.cc:1939] T 153b865087e6438eae3b0b7ed8f66dbd P 4aa027a05a704601aed873680d11ed97: Deleting consensus metadata
I20260619 20:55:22.186406 5497 tablet_replica.cc:330] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: stopping tablet replica
I20260619 20:55:22.186543 5497 raft_consensus.cc:2229] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:22.186677 5497 raft_consensus.cc:2258] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:22.186872 4996 catalog_manager.cc:4841] TS 4aa027a05a704601aed873680d11ed97 (127.0.98.2:35899): tablet 153b865087e6438eae3b0b7ed8f66dbd (table test-table [id=993be8bca46a4662b676bf29a2759049]) successfully deleted
I20260619 20:55:22.187163 5497 ts_tablet_manager.cc:1905] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:22.189066 5497 ts_tablet_manager.cc:1918] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:22.189150 5497 log.cc:1196] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/153b865087e6438eae3b0b7ed8f66dbd
I20260619 20:55:22.189395 5497 ts_tablet_manager.cc:1939] T 153b865087e6438eae3b0b7ed8f66dbd P 59ab1a05b2f040e5a74c07287b3cb874: Deleting consensus metadata
I20260619 20:55:22.189798 4994 catalog_manager.cc:4841] TS 59ab1a05b2f040e5a74c07287b3cb874 (127.0.98.3:42111): tablet 153b865087e6438eae3b0b7ed8f66dbd (table test-table [id=993be8bca46a4662b676bf29a2759049]) successfully deleted
W20260619 20:55:22.189934 4996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43601 (error 108)
I20260619 20:55:22.190429 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.190305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43601 (local address 127.0.0.1:60190)
0619 20:55:22.190360 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.190406 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:55:22.190461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5200
W20260619 20:55:22.190547 4996 catalog_manager.cc:4584] TS 90a9272d1fb547e29b65e4a6235dac3c (127.0.98.1:43601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 153b865087e6438eae3b0b7ed8f66dbd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43601: connect: Connection refused (error 111)
I20260619 20:55:22.195660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5331
I20260619 20:55:22.199824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 4978
2026-06-19T20:55:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (828 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 20:55:22.219040 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:55:22.220073 392 webserver.cc:460] Webserver started at https://127.0.0.1:44671/ using document root <none> and password file <none>
I20260619 20:55:22.220191 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:55:22.220364 392 webserver.cc:460] Webserver started at http://127.0.0.1:36339/ using document root <none> and password file <none>
2026-06-19T20:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:22Z Disabled control of system clock
I20260619 20:55:22.227723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41171
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41171
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/testchainca.cert with env {}
W20260619 20:55:22.305979 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:22.306200 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:22.306288 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:22.307433 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:22.307468 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:22.307482 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:22.307493 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:22.307505 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:55:22.307518 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:55:22.307528 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44671/jwks.json
I20260619 20:55:22.308982 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:22.309182 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:22.309342 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:55:22.314157 5505 server_base.cc:1034] running on GCE node
W20260619 20:55:22.314136 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.314131 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.314139 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:22.314702 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:22.314985 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:22.316123 5505 hybrid_clock.cc:648] HybridClock initialized: now 1781902522316113 us; error 31 us; skew 500 ppm
I20260619 20:55:22.317363 5505 webserver.cc:460] Webserver started at http://127.0.98.62:44873/ using document root <none> and password file <none>
I20260619 20:55:22.317617 5505 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:22.317665 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:22.317811 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:22.318915 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "d3cc67b1907c4400a677937bc30216f7"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "6286402746905ace4bef04fcbfe5d93e"
server_key_iv: "7239fa3bbc58a242ada5c64e1cc134f0"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.319273 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3cc67b1907c4400a677937bc30216f7"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "6286402746905ace4bef04fcbfe5d93e"
server_key_iv: "7239fa3bbc58a242ada5c64e1cc134f0"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.320616 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:55:22.321405 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:22.321607 5505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:22.321676 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "d3cc67b1907c4400a677937bc30216f7"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "6286402746905ace4bef04fcbfe5d93e"
server_key_iv: "7239fa3bbc58a242ada5c64e1cc134f0"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.321764 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:22.334568 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:22.334885 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:22.335039 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:22.338953 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41171
I20260619 20:55:22.339008 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41171 every 8 connection(s)
I20260619 20:55:22.339323 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:22.340302 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:22.341557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5505
I20260619 20:55:22.341686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:22.341858 392 external_mini_cluster.cc:1428] Setting key 48ac6a0d6cba70e461c52ed695cff314
I20260619 20:55:22.342980 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7: Bootstrap starting.
I20260619 20:55:22.343709 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:22.344038 5572 log.cc:824] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:22.344743 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7: No bootstrap required, opened a new log
I20260619 20:55:22.346879 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } }
I20260619 20:55:22.347064 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:22.347114 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cc67b1907c4400a677937bc30216f7, State: Initialized, Role: FOLLOWER
I20260619 20:55:22.347244 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } }
I20260619 20:55:22.347335 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:22.347368 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:22.347417 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.348042 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.342528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.98.62:41171)
0619 20:55:22.342743 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.342753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.342784 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:55:22.343349 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.343360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.343445 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.343586 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.343790 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.345117 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.345873 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.346585 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.346716 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.347366 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.347369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.347370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.347574 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.347615 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.347629 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.347698 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.347764 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.347840 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260619 20:55:22.348117 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } }
I20260619 20:55:22.348207 5572 leader_election.cc:302] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3cc67b1907c4400a677937bc30216f7; no voters:
I20260619 20:55:22.348539 5572 leader_election.cc:288] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:22.348694 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:22.348857 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [term 1 LEADER]: Becoming Leader. State: Replica: d3cc67b1907c4400a677937bc30216f7, State: Running, Role: LEADER
I20260619 20:55:22.348894 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:22.348934 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:55:22.349504 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } }
I20260619 20:55:22.349893 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3cc67b1907c4400a677937bc30216f7. Latest consensus state: current_term: 1 leader_uuid: "d3cc67b1907c4400a677937bc30216f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } } }
I20260619 20:55:22.349992 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.350028 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3cc67b1907c4400a677937bc30216f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cc67b1907c4400a677937bc30216f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41171 } } }
I20260619 20:55:22.350100 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.350240 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:22.350811 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:22.352336 5591 catalog_manager.cc:1347] Generated new cluster ID: abc52e8d3a5d482597c5239571070742
I20260619 20:55:22.352375 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:22.365617 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:22.366125 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:22.376590 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3cc67b1907c4400a677937bc30216f7: Generated new TSK 0
I20260619 20:55:22.376833 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:55:22.401994 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 20:55:22.402086 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:55:22.402328 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.386466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.0.98.62:41171)
0619 20:55:22.386604 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.386607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.386762 (+ 155us) server_negotiation.cc:407] Connection header received
0619 20:55:22.386896 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.386897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.386910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.386983 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:55:22.387269 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.387899 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.389082 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.389397 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.389468 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.389724 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:55:22.402110 (+ 12386us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:55:22.402278 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53208: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":113}
W20260619 20:55:22.402379 5575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53208: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 20:55:22.402971 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:22.386411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41171 (local address 127.0.0.1:53208)
0619 20:55:22.386699 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.386711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:22.386792 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:22.387046 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:22.387047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:22.387049 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:55:22.387120 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.387122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.387988 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.387989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.388964 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.388966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.389475 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.389494 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.389562 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.389585 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:55:22.402519 (+ 12934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:55:22.402650 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 20:55:22.403323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5505
2026-06-19T20:55:22Z chronyd exiting
I20260619 20:55:22.440868 392 test_util.cc:175] -----------------------------------------------
I20260619 20:55:22.440969 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902497118594-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 20:55:22.441869 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:55:22.442627 392 webserver.cc:460] Webserver started at https://127.0.0.1:40979/ using document root <none> and password file <none>
I20260619 20:55:22.442667 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:55:22.442822 392 webserver.cc:460] Webserver started at http://127.0.0.1:42743/ using document root <none> and password file <none>
2026-06-19T20:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:22Z Disabled control of system clock
I20260619 20:55:22.450153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36999
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36999
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40979/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/testchainca.cert with env {}
W20260619 20:55:22.527967 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:22.528162 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:22.528183 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:22.529317 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:22.529358 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:22.529371 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:22.529407 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:22.529435 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:55:22.529455 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:55:22.529481 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40979/jwks.json
I20260619 20:55:22.531028 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40979/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:22.531260 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:22.531466 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:22.536074 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.536075 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:22.536108 5608 server_base.cc:1034] running on GCE node
W20260619 20:55:22.536075 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:22.536482 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:22.536757 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:22.537894 5608 hybrid_clock.cc:648] HybridClock initialized: now 1781902522537873 us; error 39 us; skew 500 ppm
I20260619 20:55:22.539122 5608 webserver.cc:460] Webserver started at http://127.0.98.62:33021/ using document root <none> and password file <none>
I20260619 20:55:22.539346 5608 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:22.539389 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:22.539548 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:22.540616 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "a8bcac6c914243c0ad50334dd132fbf4"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "5715cc0cd718290838f3e8ca55fe6a81"
server_key_iv: "e3d1488deefe96179d266321e7ba6a10"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.540952 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8bcac6c914243c0ad50334dd132fbf4"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "5715cc0cd718290838f3e8ca55fe6a81"
server_key_iv: "e3d1488deefe96179d266321e7ba6a10"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.542184 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:22.542955 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:22.543110 5608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:22.543203 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "a8bcac6c914243c0ad50334dd132fbf4"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "5715cc0cd718290838f3e8ca55fe6a81"
server_key_iv: "e3d1488deefe96179d266321e7ba6a10"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.543285 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:22.557129 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:22.557360 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:22.557500 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:22.561280 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36999
I20260619 20:55:22.561313 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36999 every 8 connection(s)
I20260619 20:55:22.561626 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:22.562498 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:22.564072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5608
I20260619 20:55:22.564189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:22.564353 392 external_mini_cluster.cc:1428] Setting key 7d3fe626fd32032212d9c2e07fd440ab
I20260619 20:55:22.565052 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: Bootstrap starting.
I20260619 20:55:22.565659 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:22.565979 5675 log.cc:824] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:22.566653 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: No bootstrap required, opened a new log
I20260619 20:55:22.568923 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } }
I20260619 20:55:22.569084 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:22.569109 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bcac6c914243c0ad50334dd132fbf4, State: Initialized, Role: FOLLOWER
I20260619 20:55:22.569244 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } }
I20260619 20:55:22.569311 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:22.569379 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:22.569415 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.569756 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.564816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.0.98.62:36999)
0619 20:55:22.565003 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.565010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.565032 (+ 22us) server_negotiation.cc:407] Connection header received
0619 20:55:22.565449 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.565455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.565513 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.565625 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.565884 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.566865 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.567907 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.568271 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.568349 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.569007 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.569010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.569010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.569264 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.569305 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.569321 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.569397 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.569501 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.569574 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260619 20:55:22.570027 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } }
I20260619 20:55:22.570122 5675 leader_election.cc:302] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8bcac6c914243c0ad50334dd132fbf4; no voters:
I20260619 20:55:22.570307 5675 leader_election.cc:288] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:22.570323 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:22.570423 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [term 1 LEADER]: Becoming Leader. State: Replica: a8bcac6c914243c0ad50334dd132fbf4, State: Running, Role: LEADER
I20260619 20:55:22.570501 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } }
I20260619 20:55:22.570529 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:22.570650 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:55:22.570804 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8bcac6c914243c0ad50334dd132fbf4. Latest consensus state: current_term: 1 leader_uuid: "a8bcac6c914243c0ad50334dd132fbf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } } }
I20260619 20:55:22.570873 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.571422 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8bcac6c914243c0ad50334dd132fbf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bcac6c914243c0ad50334dd132fbf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36999 } } }
I20260619 20:55:22.571491 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4 [sys.catalog]: This master's current role is: LEADER
W20260619 20:55:22.572180 5696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:55:22.572239 5696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:55:22.572291 5684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:22.572455 5684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:22.573820 5684 catalog_manager.cc:1347] Generated new cluster ID: f120bab088034908abe3cd042d046b4a
I20260619 20:55:22.573861 5684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:22.581764 5684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:22.582296 5684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:22.586251 5684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8bcac6c914243c0ad50334dd132fbf4: Generated new TSK 0
I20260619 20:55:22.586419 5684 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:55:22.614599 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 20:55:22.614667 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:55:22.614912 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.593255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.98.62:36999)
0619 20:55:22.593308 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.593311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.593417 (+ 106us) server_negotiation.cc:407] Connection header received
0619 20:55:22.593500 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.593501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.593513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.593579 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:55:22.593736 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.594252 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.595033 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.595252 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.595338 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.595515 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:55:22.614738 (+ 19223us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:55:22.614866 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 20:55:22.614975 5678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 20:55:22.615052 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:22.593192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36999 (local address 127.0.0.1:42606)
0619 20:55:22.593341 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.593351 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:22.593434 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:22.593579 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:22.593580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:22.593582 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:55:22.593645 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.593647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.594364 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.594365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.594944 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.594946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.595339 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.595343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.595407 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.595412 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:55:22.614869 (+ 19457us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:55:22.614960 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260619 20:55:22.615406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5608
2026-06-19T20:55:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 20:55:22.652671 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:55:22.653286 392 webserver.cc:460] Webserver started at https://127.0.0.1:44009/ using document root <none> and password file <none>
I20260619 20:55:22.653324 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:55:22.653458 392 webserver.cc:460] Webserver started at http://127.0.0.1:39239/ using document root <none> and password file <none>
2026-06-19T20:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:22Z Disabled control of system clock
I20260619 20:55:22.661041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35713
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44009/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:55:22.737905 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:22.738092 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:22.738113 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:22.739259 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:22.739300 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:22.739313 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:22.739326 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:22.739339 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:55:22.739351 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:55:22.739363 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44009/jwks.json
I20260619 20:55:22.740610 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44009/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:22.740772 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:22.740983 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:22.745527 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.745576 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.745702 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:22.745704 5711 server_base.cc:1034] running on GCE node
I20260619 20:55:22.745988 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:22.746268 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:22.747443 5711 hybrid_clock.cc:648] HybridClock initialized: now 1781902522747430 us; error 29 us; skew 500 ppm
I20260619 20:55:22.748580 5711 webserver.cc:460] Webserver started at http://127.0.98.62:37491/ using document root <none> and password file <none>
I20260619 20:55:22.748837 5711 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:22.748906 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:22.749055 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:22.750051 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "16b0d691928149a5a65949666af7fb10"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "e4583d1bee139b34deaa0565fb7506f7"
server_key_iv: "c6fa5378f54dd037515374163814377b"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.750438 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "16b0d691928149a5a65949666af7fb10"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "e4583d1bee139b34deaa0565fb7506f7"
server_key_iv: "c6fa5378f54dd037515374163814377b"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.751761 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:22.752544 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:22.752722 5711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:22.752820 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "16b0d691928149a5a65949666af7fb10"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "e4583d1bee139b34deaa0565fb7506f7"
server_key_iv: "c6fa5378f54dd037515374163814377b"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.752900 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:22.764861 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:22.765146 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:22.765293 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:22.769208 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35713
I20260619 20:55:22.769241 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35713 every 8 connection(s)
I20260619 20:55:22.769560 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:22.770176 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:22.772720 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10: Bootstrap starting.
I20260619 20:55:22.773223 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:22.773450 5778 log.cc:824] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:22.774060 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10: No bootstrap required, opened a new log
I20260619 20:55:22.774896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5711
I20260619 20:55:22.775007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:22.775187 392 external_mini_cluster.cc:1428] Setting key ce721731c439b11ef4802f4fd15f2cdd
I20260619 20:55:22.776381 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } }
I20260619 20:55:22.776592 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:22.776635 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b0d691928149a5a65949666af7fb10, State: Initialized, Role: FOLLOWER
I20260619 20:55:22.776783 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } }
I20260619 20:55:22.776865 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:22.776886 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:22.776949 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.777446 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } }
I20260619 20:55:22.777531 5778 leader_election.cc:302] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b0d691928149a5a65949666af7fb10; no voters:
I20260619 20:55:22.777700 5778 leader_election.cc:288] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:22.777745 5783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:22.777876 5783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [term 1 LEADER]: Becoming Leader. State: Replica: 16b0d691928149a5a65949666af7fb10, State: Running, Role: LEADER
I20260619 20:55:22.777953 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:22.777952 5783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } }
I20260619 20:55:22.778282 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b0d691928149a5a65949666af7fb10. Latest consensus state: current_term: 1 leader_uuid: "16b0d691928149a5a65949666af7fb10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } } }
I20260619 20:55:22.778360 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.778299 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b0d691928149a5a65949666af7fb10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b0d691928149a5a65949666af7fb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35713 } } }
I20260619 20:55:22.778499 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.778584 5787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:22.779055 5787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:22.780951 5787 catalog_manager.cc:1347] Generated new cluster ID: ab7fe2d08f944330994afc5a566073a1
I20260619 20:55:22.781005 5787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:22.781505 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.775631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.0.98.62:35713)
0619 20:55:22.775834 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.775841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.775864 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:55:22.776313 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.776319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.776373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.776481 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.776744 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.778309 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.779425 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.780000 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.780450 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.780863 (+ 413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.780867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.780867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.781100 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.781131 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.781141 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.781218 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.781288 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.781361 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260619 20:55:22.782480 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:55:22.795301 5787 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:22.795806 5787 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:22.800719 5787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b0d691928149a5a65949666af7fb10: Generated new TSK 0
I20260619 20:55:22.800915 5787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:22.817836 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.814540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.0.98.62:35713)
0619 20:55:22.814601 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.814603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.814705 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:55:22.814780 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.814781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.814793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.814873 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:55:22.815002 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.815499 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.816326 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.816526 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.816602 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.816786 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:55:22.817645 (+ 859us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:55:22.817774 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 20:55:22.817888 5782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 20:55:22.817960 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:22.814467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35713 (local address 127.0.0.1:36786)
0619 20:55:22.814642 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:22.814651 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:22.814721 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:22.814873 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:22.814874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:22.814876 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:55:22.814933 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.814936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.815616 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.815618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.816217 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:22.816219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.816600 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.816607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:22.816666 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.816670 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:55:22.817753 (+ 1083us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:55:22.817855 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260619 20:55:22.818593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5711
E20260619 20:55:22.828579 5808 webserver.cc:533] Webserver: sslize failed:
E20260619 20:55:22.828634 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T20:55:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (203 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T20:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:22Z Disabled control of system clock
I20260619 20:55:22.863216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34431
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/testchainca.cert with env {}
W20260619 20:55:22.940936 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:22.941120 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:22.941140 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:22.942327 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:22.942366 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:22.942379 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:22.942391 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:22.942430 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:55:22.942458 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:55:22.942479 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 20:55:22.943908 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:22.944146 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:22.944360 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:22.949111 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.949097 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:22.949088 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:22.949364 5812 server_base.cc:1034] running on GCE node
I20260619 20:55:22.949610 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:22.949910 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:22.951068 5812 hybrid_clock.cc:648] HybridClock initialized: now 1781902522951046 us; error 37 us; skew 500 ppm
I20260619 20:55:22.952307 5812 webserver.cc:460] Webserver started at http://127.0.98.62:39643/ using document root <none> and password file <none>
I20260619 20:55:22.952536 5812 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:22.952612 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:22.952756 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:22.953785 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "f5242bfdcd5c45bc808a03336e137ec5"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "ce78ef89e78b53f9bbf014a1c8279b23"
server_key_iv: "5d3d44c2041056cf7e199dfcf8730b17"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.954126 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5242bfdcd5c45bc808a03336e137ec5"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "ce78ef89e78b53f9bbf014a1c8279b23"
server_key_iv: "5d3d44c2041056cf7e199dfcf8730b17"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.955363 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:22.956069 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:22.956256 5812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:22.956350 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "f5242bfdcd5c45bc808a03336e137ec5"
format_stamp: "Formatted at 2026-06-19 20:55:22 on dist-test-slave-8bgz"
server_key: "ce78ef89e78b53f9bbf014a1c8279b23"
server_key_iv: "5d3d44c2041056cf7e199dfcf8730b17"
server_key_version: "encryptionkey@0"
I20260619 20:55:22.956427 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:22.972286 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:22.972579 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:22.972738 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:22.976572 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34431
I20260619 20:55:22.976617 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34431 every 8 connection(s)
I20260619 20:55:22.976958 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:22.977232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5812
I20260619 20:55:22.977349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:22.977542 392 external_mini_cluster.cc:1428] Setting key e452c5a3cda179d391da3e8be20db109
I20260619 20:55:22.978276 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:22.981302 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5: Bootstrap starting.
I20260619 20:55:22.981987 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:22.982340 5879 log.cc:824] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:22.983178 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5: No bootstrap required, opened a new log
I20260619 20:55:22.983618 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:22.978055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.98.62:34431)
0619 20:55:22.978323 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:22.978332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:22.978372 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:55:22.978963 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:22.978972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:22.979052 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:22.979203 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:22.979360 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.980679 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.981435 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:22.982080 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:22.982238 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:22.982933 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:22.982937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:22.982938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:22.983136 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:22.983176 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:22.983190 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:22.983257 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:22.983335 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:55:22.983408 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260619 20:55:22.984555 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:55:22.985342 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260619 20:55:22.985496 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:22.985543 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5242bfdcd5c45bc808a03336e137ec5, State: Initialized, Role: FOLLOWER
I20260619 20:55:22.985687 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260619 20:55:22.985765 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:22.985788 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:22.985819 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:22.986351 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260619 20:55:22.986433 5879 leader_election.cc:302] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5242bfdcd5c45bc808a03336e137ec5; no voters:
I20260619 20:55:22.986538 5879 leader_election.cc:288] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:22.986629 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:22.986785 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [term 1 LEADER]: Becoming Leader. State: Replica: f5242bfdcd5c45bc808a03336e137ec5, State: Running, Role: LEADER
I20260619 20:55:22.986814 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:22.986869 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } }
I20260619 20:55:22.987195 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5242bfdcd5c45bc808a03336e137ec5. Latest consensus state: current_term: 1 leader_uuid: "f5242bfdcd5c45bc808a03336e137ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260619 20:55:22.987214 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5242bfdcd5c45bc808a03336e137ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5242bfdcd5c45bc808a03336e137ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34431 } } }
I20260619 20:55:22.987306 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.987397 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:22.987655 5891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:22.988161 5891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:22.989843 5891 catalog_manager.cc:1347] Generated new cluster ID: 3260cd72103441a98f7c581cf255f9dc
I20260619 20:55:22.989897 5891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:23.005292 5891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:23.005980 5891 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:23.013449 5891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5242bfdcd5c45bc808a03336e137ec5: Generated new TSK 0
I20260619 20:55:23.013618 5891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:23.077415 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:23.026149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.0.98.62:34431)
0619 20:55:23.026245 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:23.026247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:23.026339 (+ 92us) server_negotiation.cc:407] Connection header received
0619 20:55:23.026414 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:23.026415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:23.026427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:23.026512 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:55:23.026655 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:23.027146 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:23.027916 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:23.028124 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:23.028213 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:23.028403 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:55:23.077083 (+ 48680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:55:23.077332 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 20:55:23.077491 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:23.026113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34431 (local address 127.0.0.1:42258)
0619 20:55:23.026267 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:23.026277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:23.026356 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:23.026522 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:23.026523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:23.026524 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:55:23.026582 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:23.026584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:23.027233 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:23.027235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:23.027832 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:23.027834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:23.028220 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:23.028226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:23.028282 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:23.028288 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:55:23.077270 (+ 48982us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:55:23.077389 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260619 20:55:23.077500 5882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 20:55:23.077804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5812
2026-06-19T20:55:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (239 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:23 dist-test-slave-8bgz krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:23 dist-test-slave-8bgz krb5kdc[5920](info): set up 2 sockets
Jun 19 20:55:23 dist-test-slave-8bgz krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260619 20:55:24.282370 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:24 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:24.394738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:24.472249 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:24.472440 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:24.472462 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:24.473572 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:24.473611 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:24.473624 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:24.473636 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:24.473692 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:24.475171 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:24.475420 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:24.475626 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:24.480250 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:24.480255 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:24.480268 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:24.480543 5933 server_base.cc:1034] running on GCE node
I20260619 20:55:24.480767 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:24.481024 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:24.482182 5933 hybrid_clock.cc:648] HybridClock initialized: now 1781902524482173 us; error 36 us; skew 500 ppm
Jun 19 20:55:24 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:24.488365 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:24.488811 5933 webserver.cc:460] Webserver started at http://127.0.98.62:38475/ using document root <none> and password file <none>
I20260619 20:55:24.488999 5933 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:24.489037 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:24.489112 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:24.490103 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "484694123f8b4a168fe2b360144e1e25"
format_stamp: "Formatted at 2026-06-19 20:55:24 on dist-test-slave-8bgz"
server_key: "380e6a081ab56bea24953db5202ed5f2"
server_key_iv: "234623d9f19de50370407893c0b62b76"
server_key_version: "encryptionkey@0"
I20260619 20:55:24.490499 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "484694123f8b4a168fe2b360144e1e25"
format_stamp: "Formatted at 2026-06-19 20:55:24 on dist-test-slave-8bgz"
server_key: "380e6a081ab56bea24953db5202ed5f2"
server_key_iv: "234623d9f19de50370407893c0b62b76"
server_key_version: "encryptionkey@0"
I20260619 20:55:24.491707 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:24.492321 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:24.492468 5933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:24.492549 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "484694123f8b4a168fe2b360144e1e25"
format_stamp: "Formatted at 2026-06-19 20:55:24 on dist-test-slave-8bgz"
server_key: "380e6a081ab56bea24953db5202ed5f2"
server_key_iv: "234623d9f19de50370407893c0b62b76"
server_key_version: "encryptionkey@0"
I20260619 20:55:24.492599 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:24.506692 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:24.508875 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:24.508980 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:24.512708 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33743
I20260619 20:55:24.512763 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33743 every 8 connection(s)
I20260619 20:55:24.513217 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:24.513850 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:24.516050 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25: Bootstrap starting.
I20260619 20:55:24.516621 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:24.516902 6001 log.cc:824] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:24.517519 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25: No bootstrap required, opened a new log
I20260619 20:55:24.518829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 5933
I20260619 20:55:24.518909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:24.519084 392 external_mini_cluster.cc:1428] Setting key 12244022309f41c00ebf179f0a04ffd8
I20260619 20:55:24.519073 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260619 20:55:24.519256 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:24.519308 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484694123f8b4a168fe2b360144e1e25, State: Initialized, Role: FOLLOWER
I20260619 20:55:24.519497 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260619 20:55:24.519603 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:24.519644 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:24.519693 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:24.520383 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260619 20:55:24.520509 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484694123f8b4a168fe2b360144e1e25; no voters:
I20260619 20:55:24.520669 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:24.520699 6006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:24.520836 6006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [term 1 LEADER]: Becoming Leader. State: Replica: 484694123f8b4a168fe2b360144e1e25, State: Running, Role: LEADER
I20260619 20:55:24.520943 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:24.520925 6006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260619 20:55:24.521255 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484694123f8b4a168fe2b360144e1e25. Latest consensus state: current_term: 1 leader_uuid: "484694123f8b4a168fe2b360144e1e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } } }
I20260619 20:55:24.521343 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:24.521533 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484694123f8b4a168fe2b360144e1e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484694123f8b4a168fe2b360144e1e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } } }
I20260619 20:55:24.521603 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:24.521708 6012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:24.522408 6012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:24.528951 6012 catalog_manager.cc:1347] Generated new cluster ID: d18f4696b4164c2496774e8b4b4b7030
I20260619 20:55:24.529007 6012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:55:24 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:24.533660 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:24.519588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.0.98.62:33743)
0619 20:55:24.520232 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:24.520241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:24.520271 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:55:24.520379 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:24.520384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:24.520423 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:24.520556 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:24.521356 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:24.526720 (+ 5364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:24.527498 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:24.527927 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:24.528038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:24.530522 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:24.530529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:24.530530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:24.530540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:24.532143 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:24.532548 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:24.532551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:24.532551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:24.532654 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:24.532881 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:24.532882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:24.532882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:24.533111 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:24.533283 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:24.533359 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:55:24.533492 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":546,"threads_started":1}
I20260619 20:55:24.542338 6012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:24.542835 6012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:24.547125 6012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484694123f8b4a168fe2b360144e1e25: Generated new TSK 0
I20260619 20:55:24.547288 6012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260619 20:55:24.920663 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.368s user 0.001s sys 0.002s
I20260619 20:55:24.921165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:25.000962 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:25.001147 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:25.001168 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:25.002364 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:25.002403 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:25.002454 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:25.003906 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:25.004145 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:25.004384 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:25.009318 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:25.009313 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:25.009569 6027 server_base.cc:1034] running on GCE node
W20260619 20:55:25.009315 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:25.009887 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:25.010125 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:25.011265 6027 hybrid_clock.cc:648] HybridClock initialized: now 1781902525011258 us; error 36 us; skew 500 ppm
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.013595 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:25.014045 6027 webserver.cc:460] Webserver started at http://127.0.98.1:41725/ using document root <none> and password file <none>
I20260619 20:55:25.014362 6027 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:25.014439 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:25.014570 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:25.015610 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c783364d34f4bac85dd40ab7a9d4fe4"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "e09c2daff4661bf34ec2efa1e2c41ccd"
server_key_iv: "2c7c160bd85d1fe2ab28df4bb7ce70b8"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.015966 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c783364d34f4bac85dd40ab7a9d4fe4"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "e09c2daff4661bf34ec2efa1e2c41ccd"
server_key_iv: "2c7c160bd85d1fe2ab28df4bb7ce70b8"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.017252 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:25.017956 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.018124 6027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:25.018246 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "8c783364d34f4bac85dd40ab7a9d4fe4"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "e09c2daff4661bf34ec2efa1e2c41ccd"
server_key_iv: "2c7c160bd85d1fe2ab28df4bb7ce70b8"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.018330 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:25.028368 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:25.030038 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:25.030193 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:25.030462 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:25.030762 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:25.030817 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.030856 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:25.030882 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.036669 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37407
I20260619 20:55:25.036722 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37407 every 8 connection(s)
I20260619 20:55:25.037000 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:25.045369 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.037683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57621 (local address 127.0.98.62:33743)
0619 20:55:25.037801 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.037803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.037900 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:55:25.038855 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.038856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.038866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.038957 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.039498 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.039984 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.040927 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.041119 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.041167 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.043059 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.043060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.043061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.043067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.044304 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.044649 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.044650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.044650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.044698 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.044894 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.044894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.044895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.044989 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.045037 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.045246 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.045304 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260619 20:55:25.045521 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.037642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.1:57621)
0619 20:55:25.037788 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:25.037826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:25.038767 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:25.039013 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:25.039017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:25.039245 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:25.039428 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.039432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.040058 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.040058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.040851 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.040852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.041186 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.041187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.041276 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.041690 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:25.041697 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:25.042975 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:25.044371 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.044376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.044377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.044587 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.044753 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.044754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.044754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.044842 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.045042 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:25.045045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:25.045145 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 20:55:25.045222 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:55:25.045327 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260619 20:55:25.045842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6027
I20260619 20:55:25.045949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:25.046053 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260619 20:55:25.046147 392 external_mini_cluster.cc:1428] Setting key cab60785de4c31d964e8c58bc8ee36e7
I20260619 20:55:25.046160 6156 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:25.046360 6156 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260619 20:55:25.046916 5965 ts_manager.cc:194] Registered new tserver with Master: 8c783364d34f4bac85dd40ab7a9d4fe4 (127.0.98.1:37407)
I20260619 20:55:25.047725 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57621
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:25.068864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:25.147231 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:25.147464 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:25.147501 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:25.148626 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:25.148695 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:25.148741 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:25.150177 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:25.150444 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:25.150674 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:25.155439 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:25.155439 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:25.155468 6162 server_base.cc:1034] running on GCE node
W20260619 20:55:25.155475 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:25.155892 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:25.156172 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:25.157325 6162 hybrid_clock.cc:648] HybridClock initialized: now 1781902525157305 us; error 34 us; skew 500 ppm
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.159701 6162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:25.160045 6162 webserver.cc:460] Webserver started at http://127.0.98.2:39847/ using document root <none> and password file <none>
I20260619 20:55:25.160210 6162 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:25.160249 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:25.160331 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:25.161237 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "e6d29a4dbafa4709b4c1c6e5a12786f0"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "d73aa7abf2f5e6cddf773bb4bf07f462"
server_key_iv: "fa7b33c25c72a5c930c8b935a2e8254e"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.161530 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e6d29a4dbafa4709b4c1c6e5a12786f0"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "d73aa7abf2f5e6cddf773bb4bf07f462"
server_key_iv: "fa7b33c25c72a5c930c8b935a2e8254e"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.162809 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:25.163440 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.163621 6162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:25.163699 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "e6d29a4dbafa4709b4c1c6e5a12786f0"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "d73aa7abf2f5e6cddf773bb4bf07f462"
server_key_iv: "fa7b33c25c72a5c930c8b935a2e8254e"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.163751 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:25.189682 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:25.191500 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:25.191605 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:25.191766 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:25.191993 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:25.192027 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.192047 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:25.192060 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.197883 6162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44499
I20260619 20:55:25.197979 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44499 every 8 connection(s)
I20260619 20:55:25.198335 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:25.203223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6162
I20260619 20:55:25.203315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:25.203532 392 external_mini_cluster.cc:1428] Setting key fd108d81d8dfcce7f55d119e952dde48
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:25.208577 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.199041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33529 (local address 127.0.98.62:33743)
0619 20:55:25.199121 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.199123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.199251 (+ 128us) server_negotiation.cc:407] Connection header received
0619 20:55:25.200314 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.200315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.200324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.200365 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.200995 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.201495 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.202630 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.202815 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.202867 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.205492 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.205493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.205494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.205499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.207006 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.207563 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.207564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.207564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.207610 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.207929 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.207930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.207930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.208029 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.208081 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.208453 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.208517 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:55:25.208909 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.198984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.2:33529)
0619 20:55:25.199123 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:25.199167 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:25.200203 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:25.200425 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:25.200428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:25.200703 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:25.200921 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.200925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.201587 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.201587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.202553 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.202556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.202892 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.202893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.202996 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.203583 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:25.203591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:25.205376 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:25.207096 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.207103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.207106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.207478 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.207680 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.207681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.207681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.207851 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.208104 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:25.208111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:25.208298 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 20:55:25.208437 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 20:55:25.208617 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260619 20:55:25.209442 6291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260619 20:55:25.209550 6291 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:25.209719 6291 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260619 20:55:25.210095 5965 ts_manager.cc:194] Registered new tserver with Master: e6d29a4dbafa4709b4c1c6e5a12786f0 (127.0.98.2:44499)
I20260619 20:55:25.210475 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33529
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:25.229329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:25.308073 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:25.308303 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:25.308338 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:25.309444 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:25.309505 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:25.309546 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:25.311003 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:25.311252 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:25.311482 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:25.316183 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:25.316255 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:25.316202 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:25.316411 6296 server_base.cc:1034] running on GCE node
I20260619 20:55:25.316680 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:25.316905 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:25.318042 6296 hybrid_clock.cc:648] HybridClock initialized: now 1781902525318023 us; error 33 us; skew 500 ppm
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.320330 6296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:25.320741 6296 webserver.cc:460] Webserver started at http://127.0.98.3:39281/ using document root <none> and password file <none>
I20260619 20:55:25.320966 6296 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:25.321056 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:25.321192 6296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:25.322240 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "42a749d4be734159ac870b1286f82c83"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "dfaa61f8a8d8ccdfff6cd3fa96f5c36c"
server_key_iv: "642dbf733c0cc1e0e4a8c333ba6e4465"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.322547 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42a749d4be734159ac870b1286f82c83"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "dfaa61f8a8d8ccdfff6cd3fa96f5c36c"
server_key_iv: "642dbf733c0cc1e0e4a8c333ba6e4465"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.323715 6296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:25.324371 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.324566 6296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:25.324652 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "42a749d4be734159ac870b1286f82c83"
format_stamp: "Formatted at 2026-06-19 20:55:25 on dist-test-slave-8bgz"
server_key: "dfaa61f8a8d8ccdfff6cd3fa96f5c36c"
server_key_iv: "642dbf733c0cc1e0e4a8c333ba6e4465"
server_key_version: "encryptionkey@0"
I20260619 20:55:25.324717 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:25.360208 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:25.361960 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:25.362112 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:25.362349 6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:25.362684 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:25.362751 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.362792 6296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:25.362818 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:25.368803 6296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36169
I20260619 20:55:25.368842 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36169 every 8 connection(s)
I20260619 20:55:25.369158 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:25.374068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6296
I20260619 20:55:25.374151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:25.374399 392 external_mini_cluster.cc:1428] Setting key f5804bd282f2e6f5d546f9d0bcdfe946
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:25.378599 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.369898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45215 (local address 127.0.98.62:33743)
0619 20:55:25.369970 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.369972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.370193 (+ 221us) server_negotiation.cc:407] Connection header received
0619 20:55:25.371113 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.371114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.371123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.371175 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.371860 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.372478 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.373422 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.373623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.373676 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.376053 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.376054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.376054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.376060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.377459 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.377794 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.377795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.377795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.377840 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.378058 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.378058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.378059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.378152 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.378238 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.378481 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.378539 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:55:25.378782 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.369894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.3:45215)
0619 20:55:25.370064 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:25.370103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:25.371025 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:25.371234 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:25.371237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:25.371479 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:25.371659 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.371663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.372568 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.372568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.373343 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:25.373344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.373693 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.373693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:25.373788 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.374453 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:25.374465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:25.375944 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:25.377525 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.377529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.377531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.377735 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.377913 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:25.377913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:25.377914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:25.378006 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:25.378239 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:25.378244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:25.378372 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 20:55:25.378454 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:55:25.378563 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260619 20:55:25.379222 6425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260619 20:55:25.379314 6425 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:25.379472 6425 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260619 20:55:25.379813 5965 ts_manager.cc:194] Registered new tserver with Master: 42a749d4be734159ac870b1286f82c83 (127.0.98.3:36169)
I20260619 20:55:25.380214 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45215
I20260619 20:55:25.386257 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/test-admin.keytab.
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.401973 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 20:55:25.403496 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:25.402782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.1:37407)
0619 20:55:25.402964 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.402968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.402977 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:25.402998 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.403000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.403012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.403065 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.403417 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35070: BlockingRecv error: recv got EOF from 127.0.0.1:35070 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
W20260619 20:55:25.403613 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.404560 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.405382 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:25.412436 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.405877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.0.98.1:37407)
0619 20:55:25.405929 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.405931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.405994 (+ 63us) server_negotiation.cc:407] Connection header received
0619 20:55:25.406055 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.406055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.406064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.406109 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.406532 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.407221 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.407894 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.408192 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.408243 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.410142 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.410148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.410148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.410154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.411525 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.411817 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.411819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.411820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.411869 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.412029 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.412030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.412030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.412136 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.412229 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.412288 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.412355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 20:55:25.414011 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:25.413616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.1:37407)
0619 20:55:25.413649 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.413651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.413689 (+ 38us) server_negotiation.cc:407] Connection header received
0619 20:55:25.413733 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.413733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.413741 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.413780 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.413982 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35090: BlockingRecv error: recv got EOF from 127.0.0.1:35090 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:55:25.414110 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.415073 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.415863 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:25.421756 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.416271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.98.1:37407)
0619 20:55:25.416307 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.416309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.416336 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:55:25.416380 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.416380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.416387 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.416432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.416711 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.417129 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.417715 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.417884 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.417931 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.419584 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.419585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.419585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.419590 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.420707 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.421045 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.421045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.421046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.421098 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.421276 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.421277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.421277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.421366 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.421466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.421624 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.421703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:55:25.423218 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:25.422698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.0.98.1:37407)
0619 20:55:25.422730 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.422732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.422799 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:55:25.422867 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.422867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.422876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.422923 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.423175 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35122: BlockingRecv error: recv got EOF from 127.0.0.1:35122 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:55:25.423300 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902497118594-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.424120 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:25.424921 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:55:25.431548 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:25.425375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.0.98.1:37407)
0619 20:55:25.425406 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:25.425408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:25.425513 (+ 105us) server_negotiation.cc:407] Connection header received
0619 20:55:25.425586 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:25.425587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:25.425594 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:25.425656 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:25.425999 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.426503 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.427244 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:25.427405 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:25.427471 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:25.429503 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:25.429504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:25.429504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:25.429509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:25.430669 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.430957 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.430958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.430958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.431002 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:25.431168 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:25.431168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:25.431169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:25.431240 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:25.431299 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:25.431382 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:55:25.431430 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 20:55:25.432091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6027
I20260619 20:55:25.436753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6162
I20260619 20:55:25.440729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6296
I20260619 20:55:25.446444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 5933
2026-06-19T20:55:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2198 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[6445](info): set up 2 sockets
Jun 19 20:55:25 dist-test-slave-8bgz krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260619 20:55:26.478624 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:26 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 20:55:26.589598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:26.668725 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:26.668985 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:26.669009 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:26.670182 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:26.670246 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:26.670261 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:26.670298 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:26.670326 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:26.671787 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:26.672029 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:26.672261 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:26.676990 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:26.677029 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:26.677153 6458 server_base.cc:1034] running on GCE node
W20260619 20:55:26.677037 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:26.677520 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:26.677845 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:26.679006 6458 hybrid_clock.cc:648] HybridClock initialized: now 1781902526679000 us; error 36 us; skew 500 ppm
Jun 19 20:55:26 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:26.681344 6458 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 20:55:26.681757 6458 webserver.cc:460] Webserver started at http://127.0.98.62:33075/ using document root <none> and password file <none>
I20260619 20:55:26.681991 6458 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:26.682060 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:26.682250 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:26.683265 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "169a75a7090d42ce9880712e8768fa1d"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "2e737d1035562ea1b8feaf7c0bc76fab"
server_key_iv: "ed483fd8d6dde634ec934b7ceb061926"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.683665 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "169a75a7090d42ce9880712e8768fa1d"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "2e737d1035562ea1b8feaf7c0bc76fab"
server_key_iv: "ed483fd8d6dde634ec934b7ceb061926"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.684938 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:55:26.685629 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:26.685806 6458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:26.685892 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "169a75a7090d42ce9880712e8768fa1d"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "2e737d1035562ea1b8feaf7c0bc76fab"
server_key_iv: "ed483fd8d6dde634ec934b7ceb061926"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.685971 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:26.705436 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:26.707327 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:26.707485 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:26.711202 6458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45283
I20260619 20:55:26.711236 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45283 every 8 connection(s)
I20260619 20:55:26.711599 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:26.712603 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:26.714114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6458
I20260619 20:55:26.714259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:26.714450 392 external_mini_cluster.cc:1428] Setting key 0459573a1f7c048b92d4855621ed4581
I20260619 20:55:26.715293 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Bootstrap starting.
I20260619 20:55:26.715970 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:26.716269 6526 log.cc:824] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:26.717028 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: No bootstrap required, opened a new log
I20260619 20:55:26.719306 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } }
I20260619 20:55:26.719468 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:26.719507 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169a75a7090d42ce9880712e8768fa1d, State: Initialized, Role: FOLLOWER
I20260619 20:55:26.719682 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } }
I20260619 20:55:26.719784 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:26.719839 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:26.719889 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:26.720382 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } }
I20260619 20:55:26.720497 6526 leader_election.cc:302] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 169a75a7090d42ce9880712e8768fa1d; no voters:
I20260619 20:55:26.720664 6526 leader_election.cc:288] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:26.720686 6531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:26.720789 6531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 LEADER]: Becoming Leader. State: Replica: 169a75a7090d42ce9880712e8768fa1d, State: Running, Role: LEADER
I20260619 20:55:26.720865 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } }
I20260619 20:55:26.720911 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:26.721169 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } }
I20260619 20:55:26.721236 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:26.721345 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169a75a7090d42ce9880712e8768fa1d. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } }
I20260619 20:55:26.721441 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:26.721586 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 20:55:26 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:26.722061 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:26.725471 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:26.714964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.98.62:45283)
0619 20:55:26.715159 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:26.715168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:26.715200 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:55:26.715750 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:26.715759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:26.715819 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:26.715973 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:26.716647 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.718051 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.718823 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.719477 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.719582 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:26.722311 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:26.722315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:26.722316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:26.722328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:26.724020 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:26.724365 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:26.724368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:26.724368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:26.724457 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:26.724686 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:26.724686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:26.724687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:26.724912 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:26.725097 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:26.725173 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:55:26.725307 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260619 20:55:26.727066 6536 catalog_manager.cc:1347] Generated new cluster ID: 11604385d63f44288c37b13e3b473f01
I20260619 20:55:26.727118 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:26.739537 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:26.740075 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:26.748051 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Generated new TSK 0
I20260619 20:55:26.748240 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 20:55:26.773728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:26.853253 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:26.853454 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:26.853477 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:26.854624 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:26.854662 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:26.854689 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:26.856124 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:26.856367 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:26.856621 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:26.861568 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:26.861617 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:26.861716 6552 server_base.cc:1034] running on GCE node
W20260619 20:55:26.861595 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:26.861980 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:26.862169 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:26.863350 6552 hybrid_clock.cc:648] HybridClock initialized: now 1781902526863331 us; error 37 us; skew 500 ppm
Jun 19 20:55:26 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:26.865715 6552 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:55:26.866133 6552 webserver.cc:460] Webserver started at http://127.0.98.1:34161/ using document root <none> and password file <none>
I20260619 20:55:26.866372 6552 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:26.866418 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:26.866505 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:26.867481 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "92d75cfaaf874e508af1d249c620244f"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "0d5dac98741d62f1af3f54864004944c"
server_key_iv: "be711dba2180ea36e631b90fa06cc586"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.867831 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92d75cfaaf874e508af1d249c620244f"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "0d5dac98741d62f1af3f54864004944c"
server_key_iv: "be711dba2180ea36e631b90fa06cc586"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.869120 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:26.869843 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:26.870008 6552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:26.870105 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "92d75cfaaf874e508af1d249c620244f"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "0d5dac98741d62f1af3f54864004944c"
server_key_iv: "be711dba2180ea36e631b90fa06cc586"
server_key_version: "encryptionkey@0"
I20260619 20:55:26.870186 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:26.905998 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:26.908413 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:26.908608 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:26.908856 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:26.909174 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:26.909229 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:26.909271 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:26.909300 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:26.914791 6552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36767
I20260619 20:55:26.914839 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36767 every 8 connection(s)
I20260619 20:55:26.915136 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:26.918910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6552
I20260619 20:55:26.919013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:26.919226 392 external_mini_cluster.cc:1428] Setting key 277786b25e3748db85157eac6a2ebe66
Jun 19 20:55:26 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:26.924338 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:26.915837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49847 (local address 127.0.98.62:45283)
0619 20:55:26.915889 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:26.915890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:26.916060 (+ 170us) server_negotiation.cc:407] Connection header received
0619 20:55:26.916524 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:26.916525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:26.916534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:26.916596 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:26.917444 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.917947 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.919024 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.919274 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.919347 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:26.921532 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:26.921534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:26.921534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:26.921541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:26.922962 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:26.923377 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:26.923378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:26.923379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:26.923439 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:26.923705 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:26.923705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:26.923706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:26.923817 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:26.923891 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:26.924163 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 20:55:26.924249 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:26.924535 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:26.915790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:49847)
0619 20:55:26.915945 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:26.915987 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:26.916441 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:26.916648 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:26.916652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:26.917180 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:26.917366 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:26.917370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.918025 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.918026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:26.918939 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:26.918941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:26.919376 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:26.919377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:26.919522 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:26.920086 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:26.920093 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:26.921436 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:26.923062 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:26.923069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:26.923071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:26.923299 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:26.923507 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:26.923508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:26.923508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:26.923628 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:26.923883 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:26.923889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:26.924027 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:55:26.924143 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 20:55:26.924297 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260619 20:55:26.925017 6681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:26.925127 6681 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:26.925328 6681 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:26.925902 6490 ts_manager.cc:194] Registered new tserver with Master: 92d75cfaaf874e508af1d249c620244f (127.0.98.1:36767)
I20260619 20:55:26.926882 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49847
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 20:55:26.944794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:27.025843 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:27.026037 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:27.026067 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:27.027170 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:27.027204 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:27.027228 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:27.028597 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:27.028766 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:27.028924 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.033669 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.033821 6686 server_base.cc:1034] running on GCE node
W20260619 20:55:27.033725 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.033672 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.034134 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.034409 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.035559 6686 hybrid_clock.cc:648] HybridClock initialized: now 1781902527035538 us; error 33 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.037907 6686 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:55:27.038370 6686 webserver.cc:460] Webserver started at http://127.0.98.2:39975/ using document root <none> and password file <none>
I20260619 20:55:27.038594 6686 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.038661 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.038806 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:27.039816 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "246a46c735424910b13b74f0e1b1ccc9"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cf28cdac89fd44df433a416af3103136"
server_key_iv: "7787725c8d17c8950442531fee230ec8"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.040164 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "246a46c735424910b13b74f0e1b1ccc9"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cf28cdac89fd44df433a416af3103136"
server_key_iv: "7787725c8d17c8950442531fee230ec8"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.041397 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:27.042183 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.042378 6686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:27.042471 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "246a46c735424910b13b74f0e1b1ccc9"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cf28cdac89fd44df433a416af3103136"
server_key_iv: "7787725c8d17c8950442531fee230ec8"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.042582 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.073300 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.075380 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:27.075548 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.075791 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:27.076135 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:27.076193 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.076239 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:27.076272 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.082170 6686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46421
I20260619 20:55:27.082244 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46421 every 8 connection(s)
I20260619 20:55:27.082544 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:27.089388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6686
I20260619 20:55:27.089524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:27.089721 392 external_mini_cluster.cc:1428] Setting key e502e786a3d76ef569106b40d93a1b1c
I20260619 20:55:27.092129 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.083215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35261 (local address 127.0.98.62:45283)
0619 20:55:27.083271 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.083273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.083476 (+ 203us) server_negotiation.cc:407] Connection header received
0619 20:55:27.083994 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.083995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.084007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.084057 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.084994 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.085506 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.086561 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.086764 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.086847 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.088979 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.088980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.088980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.088988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.090699 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.091114 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.091115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.091115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.091174 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.091419 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.091420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.091420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.091555 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.091624 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.091912 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.092000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:55:27.092152 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.083197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:35261)
0619 20:55:27.083367 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.083401 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.083893 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.084152 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.084156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.084711 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.084900 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.084905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.085597 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.085598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.086467 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.086470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.086855 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.086856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.086975 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.087474 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.087482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.088871 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.090791 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.090797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.090799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.091017 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.091239 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.091240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.091240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.091352 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.091652 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.091658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.091776 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.091870 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.091988 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260619 20:55:27.092587 6815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:27.092679 6815 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:27.092844 6815 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:27.093255 6490 ts_manager.cc:194] Registered new tserver with Master: 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:27.093797 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35261
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:55:27.113250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:27.191255 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:27.191447 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:27.191468 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:27.192514 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:27.192548 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:27.192574 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:27.193909 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:27.194082 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:27.194260 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.199028 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.199088 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.199225 6820 server_base.cc:1034] running on GCE node
W20260619 20:55:27.199106 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.199501 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.199692 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.200842 6820 hybrid_clock.cc:648] HybridClock initialized: now 1781902527200831 us; error 30 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.203353 6820 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:55:27.203799 6820 webserver.cc:460] Webserver started at http://127.0.98.3:45217/ using document root <none> and password file <none>
I20260619 20:55:27.204006 6820 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.204049 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.204197 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:27.205193 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "ff4eb0d8837967e0d4ef136f504d11de"
server_key_iv: "0be2eea46e9c26f8e61007ee8f4e2133"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.205533 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "ff4eb0d8837967e0d4ef136f504d11de"
server_key_iv: "0be2eea46e9c26f8e61007ee8f4e2133"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.206862 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:27.207511 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.207695 6820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:27.207798 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "ff4eb0d8837967e0d4ef136f504d11de"
server_key_iv: "0be2eea46e9c26f8e61007ee8f4e2133"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.207867 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.233644 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.235540 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:27.235711 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.235954 6820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:27.236258 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:27.236311 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.236354 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:27.236382 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.241961 6820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45401
I20260619 20:55:27.242008 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45401 every 8 connection(s)
I20260619 20:55:27.242401 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:27.247642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6820
I20260619 20:55:27.247740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:27.247912 392 external_mini_cluster.cc:1428] Setting key d5649af2a9534dcafec539457a673bf4
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:27.251675 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.243132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59867 (local address 127.0.98.62:45283)
0619 20:55:27.243194 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.243196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.243418 (+ 222us) server_negotiation.cc:407] Connection header received
0619 20:55:27.243870 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.243871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.243879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.243940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.244816 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.245399 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.246448 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.246628 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.246684 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.248877 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.248879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.248879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.248886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.250316 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.250747 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.250749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.250749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.250807 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.251056 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.251057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.251057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.251175 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.251250 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.251513 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.251586 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:55:27.251868 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.243129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:59867)
0619 20:55:27.243286 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.243337 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.243795 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.243995 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.243998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.244529 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.244743 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.244748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.245483 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.245484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.246383 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.246385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.246702 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.246702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.246795 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.247200 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.247208 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.248781 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.250410 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.250417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.250419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.250647 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.250873 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.250874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.250874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.250985 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.251242 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.251248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.251381 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.251482 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.251631 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260619 20:55:27.252344 6949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:27.252421 6949 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:27.252614 6949 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:27.252987 6490 ts_manager.cc:194] Registered new tserver with Master: 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401)
I20260619 20:55:27.253363 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59867
I20260619 20:55:27.259574 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 20:55:27.284878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41983
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45283,127.0.98.61:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:27.363122 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:27.363313 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:27.363334 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:27.364454 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:27.364493 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:27.364507 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:27.364548 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:27.364575 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:27.365975 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45283,127.0.98.61:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
W20260619 20:55:27.366097 6954 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45283,127.0.98.61:41983'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 20:55:27.366273 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:27.366503 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.371073 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.371134 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.371053 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.371390 6954 server_base.cc:1034] running on GCE node
I20260619 20:55:27.371656 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.371932 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.373070 6954 hybrid_clock.cc:648] HybridClock initialized: now 1781902527373040 us; error 49 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.375535 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 20:55:27.375996 6954 webserver.cc:460] Webserver started at http://127.0.98.61:39407/ using document root <none> and password file <none>
I20260619 20:55:27.376222 6954 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.376288 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.376436 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:27.377453 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/instance:
uuid: "656d00a1cbcc44148dae9b1596e91cfe"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "0377f00997d55daf4be0651854fc3891"
server_key_iv: "dcc7e2ba266cde0a34416bf70a75dcd0"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.377831 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal/instance:
uuid: "656d00a1cbcc44148dae9b1596e91cfe"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "0377f00997d55daf4be0651854fc3891"
server_key_iv: "dcc7e2ba266cde0a34416bf70a75dcd0"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.379168 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:27.379882 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.380051 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.380146 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "656d00a1cbcc44148dae9b1596e91cfe"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "0377f00997d55daf4be0651854fc3891"
server_key_iv: "dcc7e2ba266cde0a34416bf70a75dcd0"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.380227 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.405822 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.407677 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:27.407838 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.412173 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41983
I20260619 20:55:27.412225 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41983 every 8 connection(s)
I20260619 20:55:27.412674 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:55:27.414147 7022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:27.416513 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:55:27.419431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 6954
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:55:27.426962 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.417406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.0.98.62:45283)
0619 20:55:27.417491 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.417493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.417726 (+ 233us) server_negotiation.cc:407] Connection header received
0619 20:55:27.418415 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.418416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.418430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.418491 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.419234 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.420009 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.421113 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.421438 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.421506 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.423893 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.423894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.423895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.423901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.425356 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.425791 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.425793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.425793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.425862 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.426144 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.426145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.426146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.426370 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.426462 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.426779 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.426857 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 20:55:27.427047 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.417367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48092)
0619 20:55:27.417541 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.417604 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.418280 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.418560 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.418564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.418907 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.419141 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.419146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.420127 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.420128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.421051 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.421054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.421530 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.421531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.421633 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.422169 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.422177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.423805 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.425432 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.425437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.425439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.425722 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.425940 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.425941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.425941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.426074 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.426461 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.426466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.426614 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.426737 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.426871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260619 20:55:27.427639 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:55:27.428607 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.419981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.61:41983)
0619 20:55:27.420264 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.420269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.420289 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:55:27.420325 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.420327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.420341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.420420 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.421049 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.422018 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.422970 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.423244 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.423295 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.425604 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.425609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.425610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.425615 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.427462 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.427830 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.427833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.427834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.427892 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.428064 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.428065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.428066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.428200 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.428336 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.428413 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.428484 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":78,"threads_started":1}
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:55:27.434275 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.427874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56840)
0619 20:55:27.427919 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.427929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.427963 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.428138 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.428138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.428268 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.428302 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.428303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.429179 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.429180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.429641 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.429642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.430024 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.430025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.430086 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.430464 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.430468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.431508 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.433290 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.433291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.433292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.433447 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.433648 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.433649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.433650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.433701 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.433924 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.433925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.433960 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.434026 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.434114 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:55:27.434315 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.427916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.98.61:41983)
0619 20:55:27.428024 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.428026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.428032 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:27.428048 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.428049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.428057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.428135 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.428378 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.429119 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.429726 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.429944 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.430024 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.431587 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.431589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.431589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.431595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.433194 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.433528 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.433529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.433530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.433585 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.433761 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.433762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.433762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.433861 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.433933 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.434055 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.434163 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260619 20:55:27.435477 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Bootstrap starting.
I20260619 20:55:27.436074 7022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:27.436335 7022 log.cc:824] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:27.436961 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: No bootstrap required, opened a new log
I20260619 20:55:27.438388 7022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } }
I20260619 20:55:27.438499 7022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:27.438521 7022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656d00a1cbcc44148dae9b1596e91cfe, State: Initialized, Role: FOLLOWER
I20260619 20:55:27.438673 7022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } }
I20260619 20:55:27.438942 7022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:27.438930 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } }
I20260619 20:55:27.439031 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:55:27.440531 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:55:27.440589 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:55:27.441988 6954 master_runner.cc:428] Detected that this master 656d00a1cbcc44148dae9b1596e91cfe is joining an existing cluster
I20260619 20:55:27.442034 6954 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41983
I20260619 20:55:27.449585 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.443121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.98.62:45283)
0619 20:55:27.443155 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.443157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.443259 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:55:27.443344 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.443346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.443356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.443404 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.443761 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.444240 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445120 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.445331 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445411 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.446446 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.446447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.446448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.446455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.448050 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.448758 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.448760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.448760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.448817 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.449047 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.449048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.449048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.449180 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.449241 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.449429 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.449497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:27.449604 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.442993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48098)
0619 20:55:27.443184 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.443195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.443254 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.443406 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.443406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.443610 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.443676 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.443677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.444319 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.444320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.445039 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.445041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.445434 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.445495 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.445913 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.445918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.446341 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.448140 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.448141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.448142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.448679 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.448911 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.448912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.448912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.448976 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.449266 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.449267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.449311 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.449404 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.449497 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260619 20:55:27.450320 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.443057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.98.61:41983)
0619 20:55:27.443089 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.443090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.443275 (+ 185us) server_negotiation.cc:407] Connection header received
0619 20:55:27.443315 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.443316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.443325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.443376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.443807 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.444571 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445549 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.445873 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445937 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.447519 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.447521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.447521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.447528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.449168 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.449516 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.449517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.449517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.449563 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.449753 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.449754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.449754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.449870 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.449940 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.450135 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.450234 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:27.450538 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.442995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56842)
0619 20:55:27.443184 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.443195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.443254 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.443409 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.443410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.443665 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.443722 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.443724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.444688 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.444689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.445482 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.445484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.445938 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.445939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.446004 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.446732 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.446736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.447434 (+ 698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.449238 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.449239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.449240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.449437 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.449622 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.449622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.449623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.449678 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.449963 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.449964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.449996 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.450380 (+ 384us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.450459 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:55:27.458513 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.452072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.98.61:41983)
0619 20:55:27.452110 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.452111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.452233 (+ 122us) server_negotiation.cc:407] Connection header received
0619 20:55:27.452324 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.452325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.452335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.452390 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.452789 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.453214 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.453956 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.454145 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.454244 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.456088 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.456089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.456090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.456096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.457339 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.457718 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.457719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.457719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.457787 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.457975 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.457976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.457976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.458095 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.458167 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.458357 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.458425 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:27.458573 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.452003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56848)
0619 20:55:27.452121 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.452166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.452257 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.452395 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.452398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.452659 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.452700 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.452702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.453289 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.453290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.453857 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.453859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.454246 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.454247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.454327 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.454751 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.454754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.456000 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.457420 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.457426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.457428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.457622 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.457850 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.457850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.457851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.457914 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.458167 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.458175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.458244 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.458332 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.458435 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260619 20:55:27.459085 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41983: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41983
}
attrs {
promote: true
}
}
dest_uuid: "169a75a7090d42ce9880712e8768fa1d"
cas_config_opid_index: -1
I20260619 20:55:27.459420 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } }
I20260619 20:55:27.459887 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.460139 6996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:27.460842 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41983
I20260619 20:55:27.460850 6996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 20:55:27.460915 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.460968 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:27.461103 6954 master.cc:537] Master@127.0.98.61:41983 shutting down...
I20260619 20:55:27.461711 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:55:27.462558 6474 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 20:55:27.462986 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d -> Peer 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61:41983): Couldn't send request to peer 656d00a1cbcc44148dae9b1596e91cfe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:55:27.463948 6954 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:27.464032 6954 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:27.464082 6954 tablet_replica.cc:330] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: stopping tablet replica
I20260619 20:55:27.476432 6954 master.cc:552] Master@127.0.98.61:41983 shutdown complete.
I20260619 20:55:27.476498 6954 master_runner.cc:305] Clearing existing system tablet
I20260619 20:55:27.476620 6954 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.476679 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.477257 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.477748 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.477869 6954 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.477926 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.477969 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "656d00a1cbcc44148dae9b1596e91cfe"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "0377f00997d55daf4be0651854fc3891"
server_key_iv: "dcc7e2ba266cde0a34416bf70a75dcd0"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.478674 6954 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:27.480214 6954 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:55:27.480271 6954 log.cc:1196] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 20:55:27.480465 6954 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Deleting consensus metadata
I20260619 20:55:27.480587 6954 master_runner.cc:315] Copying system tablet from 127.0.98.62:45283
I20260619 20:55:27.481230 6954 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45283
I20260619 20:55:27.487500 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.481590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.98.62:45283)
0619 20:55:27.481628 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.481631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.482028 (+ 397us) server_negotiation.cc:407] Connection header received
0619 20:55:27.482108 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.482109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.482120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.482171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.482565 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.483064 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.483805 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.484002 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.484094 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.485028 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.485030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.485030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.485037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.486362 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.486714 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.486715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.486716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.486773 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.486966 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.486967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.486967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.487074 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.487149 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.487324 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.487408 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:27.487516 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.481843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48102)
0619 20:55:27.481955 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.481964 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.482044 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.482181 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.482181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.482422 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.482473 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.482475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.483145 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.483146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.483720 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.483722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.484095 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.484096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.484176 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.484640 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.484644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.484940 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.486452 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.486453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.486454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.486624 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.486838 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.486839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.486839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.486895 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.487140 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.487141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.487192 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.487297 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.487409 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260619 20:55:27.488056 6510 tablet_copy_service.cc:140] P 169a75a7090d42ce9880712e8768fa1d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 656d00a1cbcc44148dae9b1596e91cfe ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48102)
I20260619 20:55:27.488160 6510 tablet_copy_service.cc:161] P 169a75a7090d42ce9880712e8768fa1d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 656d00a1cbcc44148dae9b1596e91cfe at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48102: session id = 656d00a1cbcc44148dae9b1596e91cfe-00000000000000000000000000000000
I20260619 20:55:27.488953 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:55:27.489521 6954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:55:27.490566 6954 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: tablet copy: Starting download of 0 data blocks...
I20260619 20:55:27.490710 6954 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: tablet copy: Starting download of 1 WAL segments...
I20260619 20:55:27.491569 6954 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:55:27.492326 6510 tablet_copy_service.cc:342] P 169a75a7090d42ce9880712e8768fa1d: Request end of tablet copy session 656d00a1cbcc44148dae9b1596e91cfe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48102
I20260619 20:55:27.492383 6510 tablet_copy_service.cc:433] P 169a75a7090d42ce9880712e8768fa1d: ending tablet copy session 656d00a1cbcc44148dae9b1596e91cfe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 656d00a1cbcc44148dae9b1596e91cfe
I20260619 20:55:27.494730 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.496016 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.496038 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.496061 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.498466 6954 server_base.cc:1034] running on GCE node
I20260619 20:55:27.498561 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.498745 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.499887 6954 hybrid_clock.cc:648] HybridClock initialized: now 1781902527499877 us; error 26 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.501312 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 20:55:27.501578 6954 webserver.cc:460] Webserver started at http://127.0.98.61:39469/ using document root <none> and password file <none>
I20260619 20:55:27.501703 6954 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.501737 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.502467 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:27.503050 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.503207 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.503262 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "656d00a1cbcc44148dae9b1596e91cfe"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "0377f00997d55daf4be0651854fc3891"
server_key_iv: "dcc7e2ba266cde0a34416bf70a75dcd0"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.503310 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.550669 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.552485 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.555473 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41983
I20260619 20:55:27.555534 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41983 every 8 connection(s)
I20260619 20:55:27.555814 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:55:27.556489 7124 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:55:27.556896 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Bootstrap starting.
I20260619 20:55:27.559180 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:55:27.559371 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: Bootstrap complete.
I20260619 20:55:27.559571 7124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } }
I20260619 20:55:27.559636 7124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 656d00a1cbcc44148dae9b1596e91cfe, State: Initialized, Role: LEARNER
I20260619 20:55:27.559661 7124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } }
I20260619 20:55:27.559850 7124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:27.559883 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.559999 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: LEARNER
I20260619 20:55:27.561295 7138 catalog_manager.cc:1259] Loaded cluster ID: 11604385d63f44288c37b13e3b473f01
I20260619 20:55:27.561352 7138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: loading cluster ID for follower catalog manager: success
I20260619 20:55:27.562440 7138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: acquiring CA information for follower catalog manager: success
I20260619 20:55:27.562723 7138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:55:27.587428 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.580886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.0.98.61:41983)
0619 20:55:27.581018 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.581025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.581032 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:27.581052 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.581052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.581061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.581131 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.581731 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.582463 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.583225 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.583506 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.583578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.584749 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.584751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.584751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.584760 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.586380 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.586709 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.586711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.586711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.586777 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.586965 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.586965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.586966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.587089 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.587182 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.587256 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.587330 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 20:55:27.611064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43567
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:27.690918 7142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:27.691115 7142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:27.691138 7142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:27.692239 7142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:27.692276 7142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:27.692289 7142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:27.692302 7142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:27.692313 7142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:27.693694 7142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:27.693859 7142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:27.694012 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.698494 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.698576 7142 server_base.cc:1034] running on GCE node
W20260619 20:55:27.698503 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.698513 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.698946 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.699306 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.700462 7142 hybrid_clock.cc:648] HybridClock initialized: now 1781902527700452 us; error 29 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.702986 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 20:55:27.703459 7142 webserver.cc:460] Webserver started at http://127.0.98.60:38983/ using document root <none> and password file <none>
I20260619 20:55:27.703686 7142 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.703752 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.703894 7142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:27.704967 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/instance:
uuid: "72b9bff20d7c4541b03c8131b96507e5"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cfe7ffd1917652be0e32b98e5d5fe2bb"
server_key_iv: "889462d0166f7150a993ffecbd2ad381"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.705320 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal/instance:
uuid: "72b9bff20d7c4541b03c8131b96507e5"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cfe7ffd1917652be0e32b98e5d5fe2bb"
server_key_iv: "889462d0166f7150a993ffecbd2ad381"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.706660 7142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:55:27.707350 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.707547 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:27.707655 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "72b9bff20d7c4541b03c8131b96507e5"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cfe7ffd1917652be0e32b98e5d5fe2bb"
server_key_iv: "889462d0166f7150a993ffecbd2ad381"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.707737 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.723240 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.725416 7142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:27.725569 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.730189 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43567
I20260619 20:55:27.730268 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43567 every 8 connection(s)
I20260619 20:55:27.730608 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:55:27.731562 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:27.732923 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:55:27.735313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7142
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:55:27.742363 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.733617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.98.62:45283)
0619 20:55:27.733685 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.733688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.733850 (+ 162us) server_negotiation.cc:407] Connection header received
0619 20:55:27.734404 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.734406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.734417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.734475 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.735054 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.735618 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.737047 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.737344 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.737413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.739472 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.739473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.739473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.739479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.741012 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.741503 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.741504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.741505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.741550 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.741787 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.741788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.741788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.741890 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.741955 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.742192 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.742279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:27.742547 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.733519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48106)
0619 20:55:27.733728 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.733774 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.734295 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.734537 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.734541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.734784 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.734952 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.734956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.735725 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.735726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.736972 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.736974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.737420 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.737420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.737509 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.737942 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.737949 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.739385 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.741104 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.741109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.741110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.741421 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.741619 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.741620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.741620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.741725 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.741957 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.741963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.742078 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.742172 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.742336 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260619 20:55:27.743089 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:55:27.744170 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.735818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.98.60:43567)
0619 20:55:27.735987 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.735991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.736007 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:55:27.736027 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.736029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.736051 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.736107 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.736798 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.737652 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.738378 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.738654 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.738745 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.741199 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.741204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.741204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.741210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.743024 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.743435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.743438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.743438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.743492 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.743691 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.743692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.743693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.743841 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.743957 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.744055 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.744112 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":119,"threads_started":1}
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:55:27.748737 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.743318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56876)
0619 20:55:27.743364 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.743373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.743406 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.743555 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.743555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.743698 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.743729 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.743731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.744608 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.744609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.745164 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.745166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.745568 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.745569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.745624 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.745931 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.745935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.746960 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.748116 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.748116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.748117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.748251 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.748371 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.748371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.748372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.748406 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.748561 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.748562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.748590 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.748636 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.748690 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:55:27.748775 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.743395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.98.61:41983)
0619 20:55:27.743436 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.743438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.743445 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:27.743467 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.743468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.743479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.743553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.743805 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.744533 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.745274 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.745489 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.745563 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.747026 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.747027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.747027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.747032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.748054 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.748302 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.748302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.748302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.748339 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.748444 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.748444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.748444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.748515 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.748553 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.748649 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.748706 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:55:27.749121 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:55:27.754690 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.749325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.0.1:41714)
0619 20:55:27.749370 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.749379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.749415 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.749521 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.749521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.749643 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.749663 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.749664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.750290 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.750291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.750735 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.750736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.751118 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.751119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.751165 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.751462 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.751466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.752461 (+ 995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.753720 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.753720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.753721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.753879 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.754068 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.754069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.754070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.754128 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.754378 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.754378 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.754412 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.754478 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.754564 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:55:27.754741 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.749399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.0.98.60:43567)
0619 20:55:27.749435 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.749437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.749443 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:27.749459 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.749460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.749468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.749518 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.749741 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.750180 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.750848 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.751049 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.751119 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.752554 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.752555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.752556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.752562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.753625 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.753956 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.753957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.753957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.754006 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.754198 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.754199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.754199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.754313 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.754385 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.754508 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.754618 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":23}
I20260619 20:55:27.755837 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Bootstrap starting.
I20260619 20:55:27.756515 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:27.757107 7210 log.cc:824] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:27.757652 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: No bootstrap required, opened a new log
I20260619 20:55:27.758903 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } }
I20260619 20:55:27.759014 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:27.759034 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b9bff20d7c4541b03c8131b96507e5, State: Initialized, Role: FOLLOWER
I20260619 20:55:27.759141 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } }
I20260619 20:55:27.759383 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:27.759380 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } } }
I20260619 20:55:27.759519 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:55:27.761238 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:55:27.761301 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:55:27.763221 7142 master_runner.cc:428] Detected that this master 72b9bff20d7c4541b03c8131b96507e5 is joining an existing cluster
I20260619 20:55:27.763268 7142 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43567
I20260619 20:55:27.771145 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.0.98.61:41983)
0619 20:55:27.764280 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.764282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.764415 (+ 133us) server_negotiation.cc:407] Connection header received
0619 20:55:27.764512 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.764513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.764522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.764551 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.764954 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.765689 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.766778 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767066 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767117 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.768262 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.768263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.768263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.768269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.769856 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.770310 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.770312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.770312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.770372 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.770564 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.770566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.770566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.770712 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.770799 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.770985 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.771055 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:55:27.771167 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56880)
0619 20:55:27.764350 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.764361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.764439 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.764610 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.764611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.764805 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.764869 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.764871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.765784 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.765785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.766684 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.766686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767200 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.767262 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.767715 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.767720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.768153 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.769945 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.769946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.769946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.770189 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.770445 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.770446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.770446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.770510 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.770827 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.770828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.770870 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.770960 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.771053 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":543,"threads_started":1}
I20260619 20:55:27.771729 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48112)
0619 20:55:27.764675 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.764692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.764773 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.764959 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.764959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.765194 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.765264 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.765266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.766145 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.766146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.767084 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.767087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767947 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.767994 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.768411 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.768415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.768788 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.770499 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.770500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.770500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.770730 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.771033 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.771034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.771034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.771116 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.771406 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.771407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.771444 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.771534 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.771625 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":22,"thread_start_us":105,"threads_started":1}
I20260619 20:55:27.771924 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.0.98.62:45283)
0619 20:55:27.764322 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.764324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.764743 (+ 419us) server_negotiation.cc:407] Connection header received
0619 20:55:27.764851 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.764852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.764863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.764895 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.765353 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.766065 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767179 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767829 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767872 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.768894 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.768895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.768896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.768903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.770403 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.770860 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.770861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.770862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.770936 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.771195 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.771196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.771196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.771319 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.771361 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.771731 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.771809 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:55:27.789503 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.774104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.0.1:41734)
0619 20:55:27.774152 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.774162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.774197 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.774433 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.774434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.774579 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.774611 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.774612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.775284 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.775285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.775837 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.775839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.776294 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.776295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.776376 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.776742 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.776745 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.777886 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.788303 (+ 10417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.788304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.788305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.788536 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.788800 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.788801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.788801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.788862 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.789159 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.789159 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.789211 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.789293 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.789403 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:55:27.789515 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.774154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.0.98.60:43567)
0619 20:55:27.774310 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.774311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.774318 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:27.774353 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.774353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.774361 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.774430 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.774697 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.775207 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.775947 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.776219 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.776293 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.777961 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.777962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.777962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.777969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.788194 (+ 10225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.788632 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.788633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.788634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.788727 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.788932 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.788933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.788933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.789047 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.789167 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.789342 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.789415 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260619 20:55:27.790335 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.0.1:41726)
0619 20:55:27.764804 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.764813 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.764872 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.765024 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.765025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.765223 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.765273 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.765274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.766143 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.766144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.767084 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.767087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767542 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.767601 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.786748 (+ 19147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.786753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.787190 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.789145 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.789145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.789146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.789335 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.789509 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.789509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.789509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.789553 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.789854 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.789854 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.789908 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.790096 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.790197 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":61,"thread_start_us":62,"threads_started":1}
I20260619 20:55:27.790354 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43567: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43567
}
attrs {
promote: true
}
}
dest_uuid: "169a75a7090d42ce9880712e8768fa1d"
cas_config_opid_index: 5
I20260619 20:55:27.790444 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.764580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.0.98.60:43567)
0619 20:55:27.764614 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.764616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.764853 (+ 237us) server_negotiation.cc:407] Connection header received
0619 20:55:27.764933 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.764933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.764941 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.764966 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.765346 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.766045 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767177 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.767444 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.767480 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.787501 (+ 20021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.787502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.787503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.787508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.789050 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.789406 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.789406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.789407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.789448 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.789632 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.789634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.789634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.789794 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.789846 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.790119 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.790373 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":17}
I20260619 20:55:27.790599 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
I20260619 20:55:27.790999 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.791275 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:27.792013 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:55:27.792045 7049 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43567
I20260619 20:55:27.792115 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.792219 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:27.792264 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:27.792626 7142 master.cc:537] Master@127.0.98.60:43567 shutting down...
I20260619 20:55:27.793481 7184 raft_consensus.cc:858] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
W20260619 20:55:27.793560 7184 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
I20260619 20:55:27.794240 7184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.794932 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.791016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56886)
0619 20:55:27.791158 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.791168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.791215 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.791386 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.791387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.791388 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:27.791426 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.791428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.792631 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.792632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.793834 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.793836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.794641 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.794642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.794716 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.794741 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.794834 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.794873 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 20:55:27.795261 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.791113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.98.61:41983)
0619 20:55:27.791160 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.791162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.791201 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:55:27.791266 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.791267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.791289 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.791357 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:27.791500 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.792183 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.793912 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.794565 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.794643 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.794731 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.795147 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.795190 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:27.795221 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.795466 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 20:55:27.795469 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 LEARNER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:55:27.795612 7236 mvcc.cc:205] Tried to move back new op lower bound from 7298672749558411264 to 7298672749448986624. Current Snapshot: MvccSnapshot[applied={T|T < 7298672749558411264 or (T in {7298672749558411264})}]
I20260619 20:55:27.795651 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 20:55:27.795789 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169a75a7090d42ce9880712e8768fa1d. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.795871 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 20:55:27.796185 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } }
I20260619 20:55:27.796187 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.796320 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 20:55:27.797258 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.797355 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:27.797339 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169a75a7090d42ce9880712e8768fa1d. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.797434 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: LEARNER
I20260619 20:55:27.797485 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: true } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.797541 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: LEARNER
W20260619 20:55:27.797821 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d -> Peer 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60:43567): Couldn't send request to peer 72b9bff20d7c4541b03c8131b96507e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:55:27.798312 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: attempting to promote NON_VOTER 656d00a1cbcc44148dae9b1596e91cfe to VOTER
I20260619 20:55:27.798499 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
I20260619 20:55:27.799124 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 LEARNER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 20:55:27.799191 7142 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 20:55:27.799286 7142 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 20:55:27.799332 7142 tablet_replica.cc:330] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: stopping tablet replica
I20260619 20:55:27.799294 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 20:55:27.799815 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
W20260619 20:55:27.800030 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d -> Peer 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60:43567): Couldn't send request to peer 72b9bff20d7c4541b03c8131b96507e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:55:27.800608 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.800678 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:27.800731 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.800776 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:27.801064 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.802129 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.802238 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:55:27.814563 7142 master.cc:552] Master@127.0.98.60:43567 shutdown complete.
I20260619 20:55:27.814604 7142 master_runner.cc:305] Clearing existing system tablet
I20260619 20:55:27.814751 7142 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.814790 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.815387 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.815869 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.815994 7142 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.816049 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.816089 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "72b9bff20d7c4541b03c8131b96507e5"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cfe7ffd1917652be0e32b98e5d5fe2bb"
server_key_iv: "889462d0166f7150a993ffecbd2ad381"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.816627 7142 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:27.817934 7142 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:55:27.817986 7142 log.cc:1196] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 20:55:27.818189 7142 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Deleting consensus metadata
I20260619 20:55:27.818424 7142 master_runner.cc:315] Copying system tablet from 127.0.98.62:45283
I20260619 20:55:27.819023 7142 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45283
I20260619 20:55:27.825418 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.819285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48120)
0619 20:55:27.819394 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:27.819403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:27.819462 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:27.819595 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:27.819595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:27.819784 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:27.819842 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.819844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.820578 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.820579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.821127 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:27.821129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.821627 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.821628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:27.821696 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.822444 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:27.822448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:27.822778 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:27.824386 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.824387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.824387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.824556 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.824790 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:27.824791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:27.824791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:27.824846 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:27.825096 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:27.825096 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:27.825140 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:55:27.825225 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:27.825326 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260619 20:55:27.825413 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.819339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.98.62:45283)
0619 20:55:27.819374 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.819377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.819448 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:55:27.819531 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.819532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.819541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.819591 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.819928 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.820500 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.821232 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.821546 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.821626 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.822861 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.822862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.822863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.822870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.824290 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.824640 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.824641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.824642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.824704 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.824917 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.824918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.824919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.825030 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.825104 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.825262 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.825329 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:27.825695 6510 tablet_copy_service.cc:140] P 169a75a7090d42ce9880712e8768fa1d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 72b9bff20d7c4541b03c8131b96507e5 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48120)
I20260619 20:55:27.825758 6510 tablet_copy_service.cc:161] P 169a75a7090d42ce9880712e8768fa1d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 72b9bff20d7c4541b03c8131b96507e5 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48120: session id = 72b9bff20d7c4541b03c8131b96507e5-00000000000000000000000000000000
I20260619 20:55:27.826359 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:55:27.826792 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:55:27.827852 7142 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: tablet copy: Starting download of 0 data blocks...
I20260619 20:55:27.827987 7142 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: tablet copy: Starting download of 1 WAL segments...
I20260619 20:55:27.828801 7142 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:55:27.829634 6510 tablet_copy_service.cc:342] P 169a75a7090d42ce9880712e8768fa1d: Request end of tablet copy session 72b9bff20d7c4541b03c8131b96507e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48120
I20260619 20:55:27.829699 6510 tablet_copy_service.cc:433] P 169a75a7090d42ce9880712e8768fa1d: ending tablet copy session 72b9bff20d7c4541b03c8131b96507e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 72b9bff20d7c4541b03c8131b96507e5
I20260619 20:55:27.831876 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:27.833268 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.833312 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:27.835129 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:27.835279 7142 server_base.cc:1034] running on GCE node
I20260619 20:55:27.835397 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:27.835541 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:27.836663 7142 hybrid_clock.cc:648] HybridClock initialized: now 1781902527836654 us; error 25 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:27.838085 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 20:55:27.838410 7142 webserver.cc:460] Webserver started at http://127.0.98.60:35275/ using document root <none> and password file <none>
I20260619 20:55:27.838531 7142 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:27.838585 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:27.839164 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.839701 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:27.839860 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:27.839920 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "72b9bff20d7c4541b03c8131b96507e5"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cfe7ffd1917652be0e32b98e5d5fe2bb"
server_key_iv: "889462d0166f7150a993ffecbd2ad381"
server_key_version: "encryptionkey@0"
I20260619 20:55:27.839975 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:27.848796 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:27.850327 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:27.853669 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43567
I20260619 20:55:27.853914 7316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43567 every 8 connection(s)
I20260619 20:55:27.854040 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:55:27.854727 7317 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:55:27.855124 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Bootstrap starting.
I20260619 20:55:27.856945 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:55:27.857143 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Bootstrap complete.
I20260619 20:55:27.857342 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
I20260619 20:55:27.857455 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72b9bff20d7c4541b03c8131b96507e5, State: Initialized, Role: LEARNER
I20260619 20:55:27.857502 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } }
I20260619 20:55:27.857666 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:27.858393 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:27.858500 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:55:27.859679 7331 catalog_manager.cc:1259] Loaded cluster ID: 11604385d63f44288c37b13e3b473f01
I20260619 20:55:27.859745 7331 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: loading cluster ID for follower catalog manager: success
I20260619 20:55:27.860913 7331 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: acquiring CA information for follower catalog manager: success
I20260619 20:55:27.861204 7331 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:55:27.910359 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:27.903954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.0.98.60:43567)
0619 20:55:27.904099 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:27.904101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:27.904107 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:27.904128 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:27.904129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:27.904138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:27.904212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:27.904814 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.905444 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.906144 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:27.906414 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:27.906493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:27.907781 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:27.907782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:27.907783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:27.907789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:27.909181 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.909564 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.909566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.909566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.909624 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:27.909858 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:27.909859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:27.909859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:27.909981 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:27.910074 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:27.910171 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:55:27.910292 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260619 20:55:27.910863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6552
I20260619 20:55:27.915249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36767
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45283
--webserver_port=34161
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:27.993784 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:27.993957 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:27.993997 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:27.995057 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:27.995090 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:27.995121 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:27.996475 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34161
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:27.996670 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:27.996846 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:55:28.001766 7333 server_base.cc:1034] running on GCE node
W20260619 20:55:28.001689 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:28.001698 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:28.001704 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:28.002076 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:28.002373 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:28.003544 7333 hybrid_clock.cc:648] HybridClock initialized: now 1781902528003531 us; error 27 us; skew 500 ppm
Jun 19 20:55:27 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:28.005868 7333 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:55:28.006397 7333 webserver.cc:460] Webserver started at http://127.0.98.1:34161/ using document root <none> and password file <none>
I20260619 20:55:28.006642 7333 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:28.006722 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:28.007947 7333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:28.008999 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.009150 7333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:28.009222 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "92d75cfaaf874e508af1d249c620244f"
format_stamp: "Formatted at 2026-06-19 20:55:26 on dist-test-slave-8bgz"
server_key: "0d5dac98741d62f1af3f54864004944c"
server_key_iv: "be711dba2180ea36e631b90fa06cc586"
server_key_version: "encryptionkey@0"
I20260619 20:55:28.009594 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:28.019508 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:28.021318 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:28.021417 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:28.021543 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:28.021867 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:28.021930 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.021970 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:28.021996 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.027782 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36767
I20260619 20:55:28.027828 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36767 every 8 connection(s)
I20260619 20:55:28.028121 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:28.029582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7333
I20260619 20:55:28.029748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6686
I20260619 20:55:28.037132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46421
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45283
--webserver_port=39975
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902527, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:28.042243 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.029247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58399 (local address 127.0.98.61:41983)
0619 20:55:28.029290 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.029291 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.029421 (+ 130us) server_negotiation.cc:407] Connection header received
0619 20:55:28.031112 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.031113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.031126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.031157 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.032177 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.032985 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034588 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.034938 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034979 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.038237 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.038239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.038239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.038246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.040512 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041047 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041111 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041410 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041563 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.041610 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.042006 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.042092 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:28.042291 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.028785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45853 (local address 127.0.98.60:43567)
0619 20:55:28.028840 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.028842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.029052 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:55:28.031111 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.031123 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.031137 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.031168 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.032185 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.032940 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034593 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.034902 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034944 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.038983 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.038984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.038984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.038992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.040829 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041383 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041523 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041738 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041856 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.041905 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.042104 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.042178 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:55:28.042291 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.029120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49455 (local address 127.0.98.62:45283)
0619 20:55:28.029337 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.029339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.029347 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:28.031111 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.031113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.031126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.031160 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.032181 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.032945 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034594 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.034902 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.034944 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.038655 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.038657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.038657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.038666 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.040869 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041383 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041447 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041733 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.041734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.041734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.041856 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.041905 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.042098 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.042174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260619 20:55:28.042907 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.028765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.98.1:45853)
0619 20:55:28.028931 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.028962 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.030989 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.031328 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.031332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.031785 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.032073 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.032078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.033074 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.033075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.034490 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.034492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.035032 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.035033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.035172 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.035858 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.035862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.038891 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.040990 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.040991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.040991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.041299 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041589 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.041590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.041653 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041939 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.041940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.041996 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.042398 (+ 402us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.042663 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":255,"thread_start_us":84,"threads_started":1}
I20260619 20:55:28.043084 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.029207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.98.1:58399)
0619 20:55:28.029330 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.029338 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.031005 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.031333 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.031334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.031799 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.032072 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.032078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.033080 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.033081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.034491 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.034494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.035053 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.035053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.035173 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.035804 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.035814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.038109 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.040627 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.040634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.040635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.040935 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041200 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.041201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.041331 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041677 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.041682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.041847 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.042370 (+ 523us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.042569 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":78,"thread_start_us":87,"threads_started":1}
I20260619 20:55:28.043673 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43567
I20260619 20:55:28.043708 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41983
I20260619 20:55:28.043812 7464 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.043812 7462 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.044008 7464 heartbeater.cc:511] Master 127.0.98.61:41983 requested a full tablet report, sending...
I20260619 20:55:28.044008 7462 heartbeater.cc:511] Master 127.0.98.60:43567 requested a full tablet report, sending...
I20260619 20:55:28.044341 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.029009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:49455)
0619 20:55:28.029116 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.029125 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.030989 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.031330 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.031332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.031790 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.032073 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.032078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.033074 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.033075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.034490 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.034492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.035031 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.035031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.035172 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.035951 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.035955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.038569 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.040947 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.040948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.040949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.041309 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041613 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.041613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.041614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.041662 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.041939 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.041940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.041984 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.042879 (+ 895us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.043760 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":167,"thread_start_us":70,"threads_started":1}
I20260619 20:55:28.044590 7088 ts_manager.cc:194] Registered new tserver with Master: 92d75cfaaf874e508af1d249c620244f (127.0.98.1:36767)
I20260619 20:55:28.044625 7281 ts_manager.cc:194] Registered new tserver with Master: 92d75cfaaf874e508af1d249c620244f (127.0.98.1:36767)
I20260619 20:55:28.045006 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:28.045063 7463 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.045135 7463 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:28.045365 6490 ts_manager.cc:194] Re-registered known tserver with Master: 92d75cfaaf874e508af1d249c620244f (127.0.98.1:36767)
I20260619 20:55:28.045740 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49455
W20260619 20:55:28.119048 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:28.119246 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:28.119266 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:28.120352 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:28.120391 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:28.120416 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:28.121829 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39975
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:28.122077 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:28.122345 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:28.127056 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:28.127195 7469 server_base.cc:1034] running on GCE node
W20260619 20:55:28.127074 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:28.127290 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:28.127529 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:28.127756 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:28.128937 7469 hybrid_clock.cc:648] HybridClock initialized: now 1781902528128928 us; error 39 us; skew 500 ppm
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:28.131345 7469 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:55:28.131824 7469 webserver.cc:460] Webserver started at http://127.0.98.2:39975/ using document root <none> and password file <none>
I20260619 20:55:28.132051 7469 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:28.132134 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:28.133375 7469 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:28.133904 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.134064 7469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:28.134155 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "246a46c735424910b13b74f0e1b1ccc9"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "cf28cdac89fd44df433a416af3103136"
server_key_iv: "7787725c8d17c8950442531fee230ec8"
server_key_version: "encryptionkey@0"
I20260619 20:55:28.134531 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:28.148998 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:28.150669 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:28.150827 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:28.151052 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:28.151345 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:28.151401 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.151436 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:28.151463 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.157148 7469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46421
I20260619 20:55:28.157198 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46421 every 8 connection(s)
I20260619 20:55:28.157488 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:28.162245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7469
I20260619 20:55:28.162405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6820
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:55:28.169358 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34203 (local address 127.0.98.62:45283)
0619 20:55:28.158633 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.158634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.158644 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:28.159125 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.159126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.159135 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.159183 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.159947 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.160496 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.161937 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.162258 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.162317 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.166029 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.166031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.166031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.166040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.167788 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168302 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168358 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168634 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168762 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.168805 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.169180 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.169263 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:28.169358 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45037 (local address 127.0.98.61:41983)
0619 20:55:28.158172 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.158174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.158426 (+ 252us) server_negotiation.cc:407] Connection header received
0619 20:55:28.158997 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.158998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.159011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.159067 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.159964 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.160747 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.161936 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.162251 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.162316 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.165489 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.165491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.165491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.165498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.168061 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168439 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168487 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168680 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168811 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.168851 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.169180 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.169261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:28.169414 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34205 (local address 127.0.98.60:43567)
0619 20:55:28.158223 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.158225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.158395 (+ 170us) server_negotiation.cc:407] Connection header received
0619 20:55:28.159058 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.159060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.159071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.159124 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.159966 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.160746 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.163408 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.163758 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.163799 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.166056 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.166058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.166058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.166066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.167984 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168547 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168606 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168805 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.168806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.168807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.168943 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.168985 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.169180 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.169272 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 20:55:28.169950 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:34203)
0619 20:55:28.158530 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.158539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.158895 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.159252 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.159253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.159646 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.159854 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.159858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.160599 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.160600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.161850 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.161852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.162381 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.162383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.162523 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.163209 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.163214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.165926 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.167878 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.167884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.167886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168216 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.168431 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.168432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168559 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.168853 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.168859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.169008 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.169449 (+ 441us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.169665 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260619 20:55:28.169946 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.98.2:45037)
0619 20:55:28.158370 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.158380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.158894 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.159570 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.159571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.159757 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.159854 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.159860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.160822 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.160822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.161850 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.161852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.162383 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.162384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.162523 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.163130 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.163137 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.165387 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.168147 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.168148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168358 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.168555 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.168557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168610 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.168893 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.168894 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.169008 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.169471 (+ 463us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.169782 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":177,"spinlock_wait_cycles":7552,"thread_start_us":133,"threads_started":1}
I20260619 20:55:28.169947 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.158098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.98.2:34205)
0619 20:55:28.158274 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.158305 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.158895 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.159177 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.159181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.159647 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.159854 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.159860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.161951 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.161952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.163309 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.163312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.163872 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.163873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.163945 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.164360 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.164365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.165919 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.168069 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.168070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168462 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.168672 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.168673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.168673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.168741 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.169027 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.169028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.169072 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.169583 (+ 511us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.169732 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260619 20:55:28.170457 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41983
I20260619 20:55:28.170562 7600 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.170745 7600 heartbeater.cc:511] Master 127.0.98.61:41983 requested a full tablet report, sending...
I20260619 20:55:28.170830 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:28.170885 7599 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.170949 7599 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:28.171037 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43567
I20260619 20:55:28.171082 7598 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.171144 7598 heartbeater.cc:511] Master 127.0.98.60:43567 requested a full tablet report, sending...
I20260619 20:55:28.171141 7088 ts_manager.cc:194] Registered new tserver with Master: 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:28.171325 6490 ts_manager.cc:194] Re-registered known tserver with Master: 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:28.171382 7281 ts_manager.cc:194] Registered new tserver with Master: 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:28.171824 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34203
I20260619 20:55:28.172016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45401
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45283
--webserver_port=45217
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:28.251540 7605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:28.251724 7605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:28.251749 7605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:28.252851 7605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:28.252892 7605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:28.252947 7605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:28.254456 7605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45217
--tserver_master_addrs=127.0.98.62:45283,127.0.98.61:41983,127.0.98.60:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:28.254700 7605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:28.254976 7605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:28.260007 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:28.260022 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:28.260084 7605 server_base.cc:1034] running on GCE node
W20260619 20:55:28.260027 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:28.260403 7605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:28.260588 7605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:28.261731 7605 hybrid_clock.cc:648] HybridClock initialized: now 1781902528261715 us; error 30 us; skew 500 ppm
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:28.264025 7605 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:55:28.264465 7605 webserver.cc:460] Webserver started at http://127.0.98.3:45217/ using document root <none> and password file <none>
I20260619 20:55:28.264719 7605 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:28.264806 7605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:28.266198 7605 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:55:28.266897 7620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.267179 7605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:28.267288 7605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
format_stamp: "Formatted at 2026-06-19 20:55:27 on dist-test-slave-8bgz"
server_key: "ff4eb0d8837967e0d4ef136f504d11de"
server_key_iv: "0be2eea46e9c26f8e61007ee8f4e2133"
server_key_version: "encryptionkey@0"
I20260619 20:55:28.267601 7605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:28.278049 7605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:28.279749 7605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:28.279919 7605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:28.280131 7605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:28.280406 7605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:28.280458 7605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.280496 7605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:28.280524 7605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:28.286345 7605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45401
I20260619 20:55:28.286409 7733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45401 every 8 connection(s)
I20260619 20:55:28.286725 7605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 20:55:28 dist-test-slave-8bgz krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902528, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:28.296241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7605
I20260619 20:55:28.297750 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54841 (local address 127.0.98.62:45283)
0619 20:55:28.287815 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.287817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.288769 (+ 952us) server_negotiation.cc:407] Connection header received
0619 20:55:28.288867 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.288868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.288878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.288921 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.289650 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290411 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.292016 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.292326 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.292368 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.294828 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.294830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.294830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.294837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.296344 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296832 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.296833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.296834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.296885 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.297081 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.297082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.297083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.297203 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.297257 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.297574 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.297665 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:28.297750 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54233 (local address 127.0.98.60:43567)
0619 20:55:28.287427 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.287430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.287613 (+ 183us) server_negotiation.cc:407] Connection header received
0619 20:55:28.288646 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.288647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.288659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.288719 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.289649 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290410 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.291961 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.292238 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.292275 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.294314 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.294316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.294316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.294324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.296130 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296674 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.296675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.296676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.296742 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.297024 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.297025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.297025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.297153 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.297194 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.297574 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.297650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:55:28.297773 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48701 (local address 127.0.98.61:41983)
0619 20:55:28.287455 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.287457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.287714 (+ 257us) server_negotiation.cc:407] Connection header received
0619 20:55:28.288653 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.288654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.288666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.288735 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.289629 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290098 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.291149 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.291342 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.291426 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.293879 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.293880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.293880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.293886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.296222 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296768 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.296769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.296770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.296831 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.297057 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.297058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.297058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.297199 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.297249 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.297574 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.297655 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:28.298609 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.98.3:54233)
0619 20:55:28.287506 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.287542 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.288546 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.288787 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.288790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.289293 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.289546 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.289552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290522 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.290523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.291862 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.291864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.292349 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.292350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.292418 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.292777 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.292781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.294234 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.296224 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.296588 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.296819 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.296948 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.297243 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.297249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.297411 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.297882 (+ 471us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.298414 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":456,"thread_start_us":97,"threads_started":1}
I20260619 20:55:28.299014 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:54841)
0619 20:55:28.288717 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.288727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.288798 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.288934 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.288934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.289304 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.289546 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.289552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290522 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.290523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.291933 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.291935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.292435 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.292437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.292492 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.292879 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.292883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.294728 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.296465 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.296750 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.296955 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.297009 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.297298 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.297299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.297411 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.298505 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.298610 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":53,"thread_start_us":55,"threads_started":1}
I20260619 20:55:28.299064 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43567
I20260619 20:55:28.299412 7734 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.299505 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45283
I20260619 20:55:28.299537 7735 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.299595 7734 heartbeater.cc:511] Master 127.0.98.60:43567 requested a full tablet report, sending...
I20260619 20:55:28.299595 7735 heartbeater.cc:511] Master 127.0.98.62:45283 requested a full tablet report, sending...
I20260619 20:55:28.299964 6490 ts_manager.cc:194] Re-registered known tserver with Master: 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401)
I20260619 20:55:28.300020 7281 ts_manager.cc:194] Registered new tserver with Master: 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401)
I20260619 20:55:28.300412 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.287352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.98.3:48701)
0619 20:55:28.287646 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.287657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.288557 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.289007 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.289007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.289262 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:28.289545 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.289550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.290199 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.290200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.291058 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.291060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.291435 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.291436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.291559 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.292019 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:28.292028 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:28.293771 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:28.296324 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.296683 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.296910 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:28.296910 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:28.296911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:28.296978 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:28.297301 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:28.297302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:28.297410 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.297885 (+ 475us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.298105 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":53,"thread_start_us":94,"threads_started":1}
I20260619 20:55:28.300424 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54841
I20260619 20:55:28.300705 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41983
I20260619 20:55:28.300762 7736 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:28.300825 7736 heartbeater.cc:511] Master 127.0.98.61:41983 requested a full tablet report, sending...
I20260619 20:55:28.301064 7088 ts_manager.cc:194] Registered new tserver with Master: 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401)
I20260619 20:55:28.306494 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.297779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.0.98.60:43567)
0619 20:55:28.297858 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.297859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.298523 (+ 664us) server_negotiation.cc:407] Connection header received
0619 20:55:28.298626 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.298627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.298637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.298705 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.299465 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.300309 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.301328 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.301617 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.301679 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.303689 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.303690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.303691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.303698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.305373 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.305706 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.305707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.305708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.305760 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.305963 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.305964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.305965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.306071 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.306175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.306311 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.306380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":50,"threads_started":1}
I20260619 20:55:28.307085 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.297401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.98.62:45283)
0619 20:55:28.297815 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.297818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.298361 (+ 543us) server_negotiation.cc:407] Connection header received
0619 20:55:28.298418 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.298419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.298428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.298481 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.299321 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.299985 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.301628 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.301863 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.301892 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.303348 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.303349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.303349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.303356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.305681 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.306084 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.306085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.306086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.306138 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.306399 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.306400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.306400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.306534 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.306603 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.306923 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.307008 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":58,"threads_started":1}
I20260619 20:55:28.308385 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.298296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.0.98.61:41983)
0619 20:55:28.298332 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.298334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.301423 (+ 3089us) server_negotiation.cc:407] Connection header received
0619 20:55:28.301484 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.301484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.301493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.301544 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.302294 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.303080 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.304063 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.304265 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.304337 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.305500 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.305501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.305502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.305509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.307113 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.307550 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.307552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.307552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.307620 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.307849 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.307850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.307850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.307975 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.308066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.308204 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.308281 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:28.308832 6490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:28.310333 6490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:28.317480 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.313731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46421 (local address 127.0.0.1:41492)
0619 20:55:28.313849 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.313859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.313908 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.314503 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.314504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.314506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.314552 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.314553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.315372 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.315373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.316213 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.316215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.317207 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.317208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.317284 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.317288 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.317344 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.317388 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20260619 20:55:28.317665 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.313767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.98.3:45401)
0619 20:55:28.314006 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.314009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.314037 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:55:28.314075 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.314077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.314094 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.314172 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.314446 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.315175 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.316542 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.317248 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.317295 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.317358 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.317558 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.317593 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":64,"threads_started":1}
I20260619 20:55:28.317732 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.313938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.0.98.2:46421)
0619 20:55:28.314107 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.314111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.314122 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:28.314158 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.314160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.314174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.314470 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.314632 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.315286 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.316304 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.316989 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.317072 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.317152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.317590 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.317631 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260619 20:55:28.317896 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.313671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.0.1:43730)
0619 20:55:28.313792 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.313802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.313851 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.314316 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.314317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.314320 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.314362 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.314363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.315283 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.315284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.316444 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.316447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.317417 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.317418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.317479 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.317482 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.317777 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.317818 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":82,"threads_started":1}
I20260619 20:55:28.318468 7532 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2353a25584f53a62a05b870417582 (DEFAULT_TABLE table=test-table [id=a6ab354b1b9c43e7a654810acbfcb0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:28.318614 7668 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2353a25584f53a62a05b870417582 (DEFAULT_TABLE table=test-table [id=a6ab354b1b9c43e7a654810acbfcb0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:28.318847 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2353a25584f53a62a05b870417582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:28.318971 7668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2353a25584f53a62a05b870417582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:28.319074 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.314145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36767 (local address 127.0.0.1:43356)
0619 20:55:28.315324 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.315334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.315379 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.315577 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.315578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.315580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.315612 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.315613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.316794 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.316794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.317847 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.317850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.318862 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.318863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.318923 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.318926 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.318976 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.319013 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":89,"threads_started":1}
I20260619 20:55:28.319352 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.314254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.98.1:36767)
0619 20:55:28.314588 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.314593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.315433 (+ 840us) server_negotiation.cc:407] Connection header received
0619 20:55:28.315476 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.315478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.315493 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.315566 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.315672 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.316711 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.317937 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.318678 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.318758 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.318825 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.319221 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.319276 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":83,"threads_started":1}
I20260619 20:55:28.319913 7396 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2353a25584f53a62a05b870417582 (DEFAULT_TABLE table=test-table [id=a6ab354b1b9c43e7a654810acbfcb0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:28.320216 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2353a25584f53a62a05b870417582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:28.322376 7762 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Bootstrap starting.
I20260619 20:55:28.322844 7763 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Bootstrap starting.
I20260619 20:55:28.323029 7762 tablet_bootstrap.cc:654] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:28.323365 7762 log.cc:824] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:28.323566 7763 tablet_bootstrap.cc:654] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:28.323689 7764 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Bootstrap starting.
I20260619 20:55:28.324216 7762 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: No bootstrap required, opened a new log
I20260619 20:55:28.324296 7762 ts_tablet_manager.cc:1397] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:28.324453 7763 log.cc:824] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:28.324528 7764 tablet_bootstrap.cc:654] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:28.324827 7764 log.cc:824] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:28.325151 7763 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: No bootstrap required, opened a new log
I20260619 20:55:28.325222 7763 ts_tablet_manager.cc:1397] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:28.325546 7764 tablet_bootstrap.cc:492] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: No bootstrap required, opened a new log
I20260619 20:55:28.325650 7764 ts_tablet_manager.cc:1397] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:28.327160 7762 raft_consensus.cc:348] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.327325 7762 raft_consensus.cc:374] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:28.327360 7762 raft_consensus.cc:729] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542d4c6b1c594a4292f1ac8dea2d2e2d, State: Initialized, Role: FOLLOWER
I20260619 20:55:28.327476 7762 consensus_queue.cc:260] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.327411 7763 raft_consensus.cc:348] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.327561 7763 raft_consensus.cc:374] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:28.327600 7763 raft_consensus.cc:729] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246a46c735424910b13b74f0e1b1ccc9, State: Initialized, Role: FOLLOWER
I20260619 20:55:28.327692 7762 ts_tablet_manager.cc:1428] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:28.327704 7763 consensus_queue.cc:260] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.327976 7763 ts_tablet_manager.cc:1428] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:28.328028 7735 heartbeater.cc:503] Master 127.0.98.62:45283 was elected leader, sending a full tablet report...
I20260619 20:55:28.328683 7599 heartbeater.cc:503] Master 127.0.98.62:45283 was elected leader, sending a full tablet report...
I20260619 20:55:28.329253 7764 raft_consensus.cc:348] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.329402 7764 raft_consensus.cc:374] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:28.329440 7764 raft_consensus.cc:729] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d75cfaaf874e508af1d249c620244f, State: Initialized, Role: FOLLOWER
I20260619 20:55:28.329530 7764 consensus_queue.cc:260] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.329795 7764 ts_tablet_manager.cc:1428] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:55:28.329954 7463 heartbeater.cc:503] Master 127.0.98.62:45283 was elected leader, sending a full tablet report...
I20260619 20:55:28.331333 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.325731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.0.1:41778)
0619 20:55:28.325772 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.325782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.325816 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.325993 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.325994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.325995 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.326025 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.326026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.326990 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.326991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.328019 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.328023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.331083 (+ 3060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.331084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.331131 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.331134 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.331219 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.331264 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 20:55:28.331575 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.325792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.60:43567)
0619 20:55:28.325830 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.325832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.325840 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:28.325866 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.325867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.325887 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.325942 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.326092 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.326896 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.330563 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.331000 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.331060 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.331102 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.331441 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.331495 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:28.331827 7291 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 20:55:28.332221 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169a75a7090d42ce9880712e8768fa1d. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:28.332317 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:55:28.356971 7770 raft_consensus.cc:482] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:28.357066 7770 raft_consensus.cc:504] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.357412 7770 leader_election.cc:288] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401), 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:28.360519 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.357646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47881 (local address 127.0.98.3:45401)
0619 20:55:28.357695 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.357698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.357707 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:28.357749 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.357750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.357761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.357849 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.358008 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.358574 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.359763 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.360173 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.360276 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.360285 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.360420 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.360462 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:55:28.360530 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.357680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47789 (local address 127.0.98.2:46421)
0619 20:55:28.357724 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.357726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.357736 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:28.357804 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.357805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.357816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.357907 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:28.358082 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.358652 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.359699 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.360156 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.360236 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.360246 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.360381 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.360443 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:55:28.360539 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.357565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.98.1:47881)
0619 20:55:28.357642 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.357654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.357698 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.357865 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.357866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.357867 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.357935 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.357942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.358669 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.358670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.359662 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.359664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.360286 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.360287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.360354 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.360356 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.360408 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.360437 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":27}
I20260619 20:55:28.360754 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.357569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46421 (local address 127.0.98.1:47789)
0619 20:55:28.357656 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:28.357666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:28.357705 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:28.357952 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:28.357953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:28.357954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:28.358000 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.358001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.358730 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.358731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.359616 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:28.359618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.360235 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.360236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:28.360307 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.360310 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:28.360367 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:55:28.360414 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":43}
I20260619 20:55:28.360883 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2353a25584f53a62a05b870417582" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" is_pre_election: true
I20260619 20:55:28.361017 7688 raft_consensus.cc:2454] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 0.
I20260619 20:55:28.361020 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2353a25584f53a62a05b870417582" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "246a46c735424910b13b74f0e1b1ccc9" is_pre_election: true
I20260619 20:55:28.361192 7552 raft_consensus.cc:2454] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 0.
I20260619 20:55:28.361260 7352 leader_election.cc:302] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542d4c6b1c594a4292f1ac8dea2d2e2d, 92d75cfaaf874e508af1d249c620244f; no voters:
I20260619 20:55:28.361374 7770 raft_consensus.cc:2781] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:28.361459 7770 raft_consensus.cc:482] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:28.361482 7770 raft_consensus.cc:3037] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:28.362278 7770 raft_consensus.cc:504] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.362424 7770 leader_election.cc:288] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 election: Requested vote from peers 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401), 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:28.362583 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2353a25584f53a62a05b870417582" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "246a46c735424910b13b74f0e1b1ccc9"
I20260619 20:55:28.362641 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2353a25584f53a62a05b870417582" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
I20260619 20:55:28.362653 7552 raft_consensus.cc:3037] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:28.362701 7688 raft_consensus.cc:3037] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:28.363899 7688 raft_consensus.cc:2454] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 1.
I20260619 20:55:28.363899 7552 raft_consensus.cc:2454] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 1.
I20260619 20:55:28.364048 7352 leader_election.cc:302] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542d4c6b1c594a4292f1ac8dea2d2e2d, 92d75cfaaf874e508af1d249c620244f; no voters:
I20260619 20:55:28.364147 7770 raft_consensus.cc:2781] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:28.364291 7770 raft_consensus.cc:686] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 1 LEADER]: Becoming Leader. State: Replica: 92d75cfaaf874e508af1d249c620244f, State: Running, Role: LEADER
I20260619 20:55:28.364385 7770 consensus_queue.cc:237] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } }
I20260619 20:55:28.365223 6490 catalog_manager.cc:5495] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d75cfaaf874e508af1d249c620244f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "92d75cfaaf874e508af1d249c620244f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } health_report { overall_health: HEALTHY } } }
I20260619 20:55:28.365725 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: true } } }
I20260619 20:55:28.365820 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:28.366583 7777 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: attempting to promote NON_VOTER 72b9bff20d7c4541b03c8131b96507e5 to VOTER
I20260619 20:55:28.366725 7777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:28.367238 7290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 LEARNER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 20:55:28.367285 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Refusing update from remote peer 169a75a7090d42ce9880712e8768fa1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 20:55:28.367486 7777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 20:55:28.367616 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 20:55:28.368171 7777 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.368399 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.368404 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.368983 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.369064 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:28.369031 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.369094 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:28.369117 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.369171 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:28.369283 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.369378 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:55:28.369403 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169a75a7090d42ce9880712e8768fa1d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:28.369470 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:55:28.377419 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.375391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.98.1:36767)
0619 20:55:28.375444 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.375446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.375455 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:28.375490 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.375491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.375504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.375566 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:28.375673 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.376113 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.376798 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.376985 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.377036 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.377160 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:28.377268 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:28.377326 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.377358 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.377373 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:28.378995 7688 raft_consensus.cc:1261] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Refusing update from remote peer 92d75cfaaf874e508af1d249c620244f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:28.379010 7552 raft_consensus.cc:1261] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Refusing update from remote peer 92d75cfaaf874e508af1d249c620244f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:28.379330 7770 consensus_queue.cc:1035] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Connected to new peer: Peer: permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:28.379463 7775 consensus_queue.cc:1035] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Connected to new peer: Peer: permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:28.381496 7781 mvcc.cc:205] Tried to move back new op lower bound from 7298672756237811712 to 7298672756180910080. Current Snapshot: MvccSnapshot[applied={T|T < 7298672756237811712}]
I20260619 20:55:28.382296 7782 mvcc.cc:205] Tried to move back new op lower bound from 7298672756237811712 to 7298672756180910080. Current Snapshot: MvccSnapshot[applied={T|T < 7298672756237811712 or (T in {7298672756237811712})}]
I20260619 20:55:28.388903 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:28.388993 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:28.390636 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 169a75a7090d42ce9880712e8768fa1d: Sending DeleteTablet for 3 replicas of tablet d3f2353a25584f53a62a05b870417582
I20260619 20:55:28.390949 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f2353a25584f53a62a05b870417582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41492
I20260619 20:55:28.391036 7668 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f2353a25584f53a62a05b870417582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43730
I20260619 20:55:28.391263 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f2353a25584f53a62a05b870417582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260619 20:55:28.391482 7793 tablet_replica.cc:330] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: stopping tablet replica
I20260619 20:55:28.391577 7793 raft_consensus.cc:2229] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:28.391655 7793 raft_consensus.cc:2258] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:28.392158 7793 ts_tablet_manager.cc:1905] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:28.392686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6458
I20260619 20:55:28.393628 7791 tablet_replica.cc:330] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: stopping tablet replica
I20260619 20:55:28.393718 7791 raft_consensus.cc:2229] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:28.393824 7791 raft_consensus.cc:2258] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:28.394270 7791 ts_tablet_manager.cc:1905] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:28.394486 7793 ts_tablet_manager.cc:1918] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:28.394550 7793 log.cc:1196] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/d3f2353a25584f53a62a05b870417582
I20260619 20:55:28.394788 7793 ts_tablet_manager.cc:1939] T d3f2353a25584f53a62a05b870417582 P 92d75cfaaf874e508af1d249c620244f: Deleting consensus metadata
I20260619 20:55:28.395882 7791 ts_tablet_manager.cc:1918] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:28.395931 7791 log.cc:1196] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/d3f2353a25584f53a62a05b870417582
I20260619 20:55:28.396106 7791 ts_tablet_manager.cc:1939] T d3f2353a25584f53a62a05b870417582 P 246a46c735424910b13b74f0e1b1ccc9: Deleting consensus metadata
I20260619 20:55:28.396324 7792 tablet_replica.cc:330] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: stopping tablet replica
I20260619 20:55:28.396461 7792 raft_consensus.cc:2229] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:28.396569 7792 raft_consensus.cc:2258] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:28.396970 7792 ts_tablet_manager.cc:1905] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:28.398130 7792 ts_tablet_manager.cc:1918] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:28.398214 7792 log.cc:1196] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/d3f2353a25584f53a62a05b870417582
I20260619 20:55:28.398460 7792 ts_tablet_manager.cc:1939] T d3f2353a25584f53a62a05b870417582 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting consensus metadata
W20260619 20:55:28.399457 7352 connection.cc:535] server connection from 127.0.0.1:43356 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 20:55:28.399577 7488 connection.cc:535] server connection from 127.0.0.1:41492 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 20:55:28.399593 7624 connection.cc:535] server connection from 127.0.0.1:43730 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:55:28.408329 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.401307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.60:43567)
0619 20:55:28.401346 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.401348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.401355 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:28.401375 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.401377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.401387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.401432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.401978 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.402739 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.403844 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.404083 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.404643 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.405420 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.405421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.405422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.405429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.407092 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.407496 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.407497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.407497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.407551 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.407764 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.407765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.407766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.407893 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.407963 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.408153 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.408241 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:28.408390 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:28.401125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.98.61:41983)
0619 20:55:28.401168 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:28.401170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:28.401284 (+ 114us) server_negotiation.cc:407] Connection header received
0619 20:55:28.401394 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:28.401395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:28.401405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:28.401453 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:28.402096 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.402800 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.403639 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:28.403887 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:28.403949 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:28.405500 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:28.405500 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:28.405501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:28.405508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:28.407264 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.407655 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.407656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.407657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.407705 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:28.407911 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:28.407912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:28.407912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:28.408048 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:28.408125 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:28.408226 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:55:28.408297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:29.384364 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.384103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:36147)
0619 20:55:29.384228 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.384296 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260619 20:55:29.384595 7735 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45283 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.388047 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.387846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:44731)
0619 20:55:29.387948 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.388011 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
W20260619 20:55:29.388228 7599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45283 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.388437 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.388226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:46145)
0619 20:55:29.388341 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.388403 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":47,"threads_started":1}
W20260619 20:55:29.388607 7463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45283 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.890569 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 169a75a7090d42ce9880712e8768fa1d)
I20260619 20:55:29.890767 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:29.891105 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 169a75a7090d42ce9880712e8768fa1d)
I20260619 20:55:29.891237 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:29.891508 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60:43567)
I20260619 20:55:29.891458 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48592)
0619 20:55:29.891362 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.891419 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260619 20:55:29.891474 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61:41983)
W20260619 20:55:29.891645 7266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.891760 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48602)
0619 20:55:29.891668 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.891720 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260619 20:55:29.892056 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48614)
0619 20:55:29.891999 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.892026 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 20:55:29.892046 7073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
W20260619 20:55:29.892118 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.892509 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.892404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48622)
0619 20:55:29.892440 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.892476 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 20:55:29.892627 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.894605 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43567 (local address 127.0.0.1:41790)
0619 20:55:29.891813 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.891823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:29.891884 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:29.892064 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:29.892065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:29.892068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:29.892227 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:29.892230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.893049 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.893050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:29.893804 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:29.893806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.894370 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.894371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:29.894442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:29.894446 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:29.894494 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:55:29.894539 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":44,"threads_started":1}
I20260619 20:55:29.894619 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.0.98.60:43567)
0619 20:55:29.891931 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:29.891933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:29.891941 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:29.891965 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:29.891966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:29.891975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:29.892040 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:29.892312 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.892946 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.893899 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.894308 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.894361 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:29.894370 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:29.894509 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 20:55:29.894553 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260619 20:55:29.894675 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.0.98.61:41983)
0619 20:55:29.892006 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:29.892008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:29.892016 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:29.892134 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:29.892135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:29.892146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:29.892221 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:29.892398 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.893012 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.893922 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.894325 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.894411 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:29.894432 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:29.894550 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 20:55:29.894580 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":48,"threads_started":1}
I20260619 20:55:29.894629 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.891237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41983 (local address 127.0.0.1:56916)
0619 20:55:29.891404 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.891417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:29.891475 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:29.892238 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:29.892239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:29.892241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:29.892288 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:29.892290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.893100 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.893101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:29.893820 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:29.893821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:29.894434 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:29.894435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:29.894471 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:29.894474 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:29.894539 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:55:29.894571 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260619 20:55:29.894821 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "656d00a1cbcc44148dae9b1596e91cfe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72b9bff20d7c4541b03c8131b96507e5" is_pre_election: true
I20260619 20:55:29.894948 7290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 656d00a1cbcc44148dae9b1596e91cfe in term 1.
I20260619 20:55:29.895113 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 656d00a1cbcc44148dae9b1596e91cfe, 72b9bff20d7c4541b03c8131b96507e5; no voters: 169a75a7090d42ce9880712e8768fa1d
I20260619 20:55:29.895146 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b9bff20d7c4541b03c8131b96507e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "656d00a1cbcc44148dae9b1596e91cfe" is_pre_election: true
I20260619 20:55:29.895220 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b9bff20d7c4541b03c8131b96507e5 in term 1.
I20260619 20:55:29.895262 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 20:55:29.895321 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 169a75a7090d42ce9880712e8768fa1d)
I20260619 20:55:29.895354 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:55:29.895362 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 656d00a1cbcc44148dae9b1596e91cfe, 72b9bff20d7c4541b03c8131b96507e5; no voters: 169a75a7090d42ce9880712e8768fa1d
I20260619 20:55:29.895469 7804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 20:55:29.895515 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 169a75a7090d42ce9880712e8768fa1d)
I20260619 20:55:29.895541 7804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:55:29.896267 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:29.896265 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:29.896395 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 election: Requested vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 72b9bff20d7c4541b03c8131b96507e5 (127.0.98.60:43567)
I20260619 20:55:29.896405 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 election: Requested vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61:41983)
I20260619 20:55:29.896564 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.896485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48624)
0619 20:55:29.896517 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.896542 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 20:55:29.896601 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "656d00a1cbcc44148dae9b1596e91cfe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72b9bff20d7c4541b03c8131b96507e5"
I20260619 20:55:29.896684 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b9bff20d7c4541b03c8131b96507e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "656d00a1cbcc44148dae9b1596e91cfe"
I20260619 20:55:29.896757 7098 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 72b9bff20d7c4541b03c8131b96507e5 in current term 2: Already voted for candidate 656d00a1cbcc44148dae9b1596e91cfe in this term.
I20260619 20:55:29.896720 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.896521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48626)
0619 20:55:29.896565 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.896594 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:55:29.896694 7290 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 656d00a1cbcc44148dae9b1596e91cfe in current term 2: Already voted for candidate 72b9bff20d7c4541b03c8131b96507e5 in this term.
I20260619 20:55:29.897047 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.896859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48640)
0619 20:55:29.896998 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.897025 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260619 20:55:29.897112 7073 leader_election.cc:334] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.897182 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:29.897106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48650)
0619 20:55:29.897136 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:29.897160 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 20:55:29.897200 7073 leader_election.cc:302] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 656d00a1cbcc44148dae9b1596e91cfe; no voters: 169a75a7090d42ce9880712e8768fa1d, 72b9bff20d7c4541b03c8131b96507e5
W20260619 20:55:29.897238 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:29.897279 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72b9bff20d7c4541b03c8131b96507e5; no voters: 169a75a7090d42ce9880712e8768fa1d, 656d00a1cbcc44148dae9b1596e91cfe
I20260619 20:55:29.897282 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 20:55:29.897372 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 20:55:30.386649 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:30.386437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:42293)
0619 20:55:30.386543 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:30.386595 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 20:55:30.388693 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:30.388508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:54367)
0619 20:55:30.388613 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:30.388651 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260619 20:55:30.389132 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:30.388883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:50097)
0619 20:55:30.389047 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:30.389086 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":59,"threads_started":1}
I20260619 20:55:31.387375 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.387129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:51945)
0619 20:55:31.387246 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.387313 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260619 20:55:31.389249 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.389092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:53479)
0619 20:55:31.389183 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.389216 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
I20260619 20:55:31.389645 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.389494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:33415)
0619 20:55:31.389574 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.389604 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":47,"threads_started":1}
I20260619 20:55:31.744534 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:31.744643 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:31.744835 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61:41983)
I20260619 20:55:31.745126 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.744959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48868)
0619 20:55:31.745057 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.745089 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260619 20:55:31.745095 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b9bff20d7c4541b03c8131b96507e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "656d00a1cbcc44148dae9b1596e91cfe" is_pre_election: true
I20260619 20:55:31.745186 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b9bff20d7c4541b03c8131b96507e5 in term 2.
I20260619 20:55:31.745348 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 656d00a1cbcc44148dae9b1596e91cfe, 72b9bff20d7c4541b03c8131b96507e5; no voters:
I20260619 20:55:31.745478 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 20:55:31.745519 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:31.745550 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:55:31.745582 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.745496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48870)
0619 20:55:31.745529 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.745559 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 20:55:31.745645 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:31.746351 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:31.746481 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 election: Requested vote from peers 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283), 656d00a1cbcc44148dae9b1596e91cfe (127.0.98.61:41983)
I20260619 20:55:31.746631 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.746550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48872)
0619 20:55:31.746582 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.746614 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 20:55:31.746621 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b9bff20d7c4541b03c8131b96507e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "656d00a1cbcc44148dae9b1596e91cfe"
I20260619 20:55:31.746702 7098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:55:31.746965 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:31.746893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48886)
0619 20:55:31.746930 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:31.746952 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 20:55:31.747072 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
I20260619 20:55:31.747509 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b9bff20d7c4541b03c8131b96507e5 in term 3.
I20260619 20:55:31.747666 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 656d00a1cbcc44148dae9b1596e91cfe, 72b9bff20d7c4541b03c8131b96507e5; no voters: 169a75a7090d42ce9880712e8768fa1d
I20260619 20:55:31.747768 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:55:31.747882 7820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [term 3 LEADER]: Becoming Leader. State: Replica: 72b9bff20d7c4541b03c8131b96507e5, State: Running, Role: LEADER
I20260619 20:55:31.747948 7820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } }
I20260619 20:55:31.748281 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b9bff20d7c4541b03c8131b96507e5. Latest consensus state: current_term: 3 leader_uuid: "72b9bff20d7c4541b03c8131b96507e5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:31.748371 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:32.210956 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [term 3 FOLLOWER]: Refusing update from remote peer 72b9bff20d7c4541b03c8131b96507e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260619 20:55:32.211243 7823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 20:55:32.211918 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b9bff20d7c4541b03c8131b96507e5. Latest consensus state: current_term: 3 leader_uuid: "72b9bff20d7c4541b03c8131b96507e5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:32.212009 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:55:32.212663 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "72b9bff20d7c4541b03c8131b96507e5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:32.212859 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656d00a1cbcc44148dae9b1596e91cfe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:55:32.212881 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.212720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48938)
0619 20:55:32.212759 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.212793 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 20:55:32.212809 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "72b9bff20d7c4541b03c8131b96507e5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "169a75a7090d42ce9880712e8768fa1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45283 } } peers { permanent_uuid: "656d00a1cbcc44148dae9b1596e91cfe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41983 } attrs { promote: false } } peers { permanent_uuid: "72b9bff20d7c4541b03c8131b96507e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43567 } attrs { promote: false } } }
I20260619 20:55:32.212932 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:32.213248 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.213146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.0.1:48946)
0619 20:55:32.213203 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.213228 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 20:55:32.213315 7266 consensus_peers.cc:487] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5 -> Peer 169a75a7090d42ce9880712e8768fa1d (127.0.98.62:45283): Couldn't send request to peer 169a75a7090d42ce9880712e8768fa1d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 20:55:32.263980 7825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:32.264483 7825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:32.264571 7825 catalog_manager.cc:1259] Loaded cluster ID: 11604385d63f44288c37b13e3b473f01
I20260619 20:55:32.264619 7825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:32.264819 7825 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:32.264904 7825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Loaded TSK: 0
I20260619 20:55:32.265057 7825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:32.317003 7280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:32.318643 7280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:32.324656 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.1:36767)
0619 20:55:32.321763 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:32.321765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:32.321773 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:32.321801 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:32.321802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:32.321813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:32.321903 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:32.322074 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322870 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.323761 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.324341 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324403 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.324414 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:32.324545 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:55:32.324582 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":67,"threads_started":1}
I20260619 20:55:32.324654 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36767 (local address 127.0.0.1:43370)
0619 20:55:32.321608 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.321620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:32.321671 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:32.321909 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:32.321910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:32.321912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:32.321962 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.321964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322973 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.322974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.323675 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.323676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.324405 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.324463 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.324466 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:32.324538 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:55:32.324584 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 20:55:32.324992 7396 tablet_service.cc:1467] Processing CreateTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 (DEFAULT_TABLE table=test-table [id=42335afa02c5473582857847da68cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:32.325084 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46421 (local address 127.0.0.1:41494)
0619 20:55:32.321780 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.321789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:32.321824 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:32.321998 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:32.321998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:32.322000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:32.322032 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.322033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322931 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.322932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.324162 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.324164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.324869 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.324930 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.324935 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:32.324986 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:55:32.325022 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260619 20:55:32.325110 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.98.2:46421)
0619 20:55:32.321808 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:32.321810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:32.321820 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:32.321878 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:32.321879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:32.321892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:32.321948 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:32.322105 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322869 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324240 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.324802 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324869 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.324880 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:32.324993 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 20:55:32.325036 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
I20260619 20:55:32.325135 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dae5c20d5c4ea0a78cb991db88bfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:32.325477 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.0.1:43732)
0619 20:55:32.321644 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.321656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:32.321706 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:32.322185 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:32.322186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:32.322188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:32.322253 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.322254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322863 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.322864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.324230 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:32.324233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.325250 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.325251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:32.325323 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.325326 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:32.325379 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:55:32.325416 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":31,"thread_start_us":83,"threads_started":1}
I20260619 20:55:32.325496 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.321640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.3:45401)
0619 20:55:32.322033 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:32.322035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:32.322043 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:32.322068 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:32.322069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:32.322079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:32.322138 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:32.322307 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.322784 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.324300 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.325140 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.325206 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.325218 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:32.325406 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 20:55:32.325443 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":59,"threads_started":1}
I20260619 20:55:32.325740 7668 tablet_service.cc:1467] Processing CreateTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 (DEFAULT_TABLE table=test-table [id=42335afa02c5473582857847da68cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:32.325830 7668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dae5c20d5c4ea0a78cb991db88bfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:32.327225 7532 tablet_service.cc:1467] Processing CreateTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 (DEFAULT_TABLE table=test-table [id=42335afa02c5473582857847da68cb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:32.327339 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dae5c20d5c4ea0a78cb991db88bfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:32.327801 7839 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: Bootstrap starting.
I20260619 20:55:32.327967 7838 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Bootstrap starting.
I20260619 20:55:32.328198 7839 tablet_bootstrap.cc:654] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:32.328380 7838 tablet_bootstrap.cc:654] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:32.328871 7842 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Bootstrap starting.
I20260619 20:55:32.328918 7838 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: No bootstrap required, opened a new log
I20260619 20:55:32.328995 7838 ts_tablet_manager.cc:1397] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:32.329178 7838 raft_consensus.cc:348] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.329272 7838 raft_consensus.cc:374] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:32.329294 7838 raft_consensus.cc:729] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542d4c6b1c594a4292f1ac8dea2d2e2d, State: Initialized, Role: FOLLOWER
I20260619 20:55:32.329294 7842 tablet_bootstrap.cc:654] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:32.329303 7839 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: No bootstrap required, opened a new log
I20260619 20:55:32.329342 7839 ts_tablet_manager.cc:1397] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:32.329448 7839 raft_consensus.cc:348] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.329525 7839 raft_consensus.cc:374] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:32.329550 7839 raft_consensus.cc:729] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d75cfaaf874e508af1d249c620244f, State: Initialized, Role: FOLLOWER
I20260619 20:55:32.329360 7838 consensus_queue.cc:260] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.329613 7839 consensus_queue.cc:260] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.329713 7838 ts_tablet_manager.cc:1428] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:55:32.329741 7839 ts_tablet_manager.cc:1428] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:32.330286 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.329943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:52283)
0619 20:55:32.330185 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.330244 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260619 20:55:32.330348 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.330145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:35169)
0619 20:55:32.330278 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.330316 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260619 20:55:32.330677 7462 heartbeater.cc:503] Master 127.0.98.60:43567 was elected leader, sending a full tablet report...
I20260619 20:55:32.330802 7734 heartbeater.cc:503] Master 127.0.98.60:43567 was elected leader, sending a full tablet report...
I20260619 20:55:32.330785 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.330694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:48509)
0619 20:55:32.330729 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.330760 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 20:55:32.331218 7842 tablet_bootstrap.cc:492] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: No bootstrap required, opened a new log
I20260619 20:55:32.331270 7842 ts_tablet_manager.cc:1397] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:32.331389 7842 raft_consensus.cc:348] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.331457 7842 raft_consensus.cc:374] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:32.331490 7842 raft_consensus.cc:729] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246a46c735424910b13b74f0e1b1ccc9, State: Initialized, Role: FOLLOWER
I20260619 20:55:32.331559 7842 consensus_queue.cc:260] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.331727 7842 ts_tablet_manager.cc:1428] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:32.332098 7598 heartbeater.cc:503] Master 127.0.98.60:43567 was elected leader, sending a full tablet report...
I20260619 20:55:32.332155 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.331938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:40123)
0619 20:55:32.332068 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.332113 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":88,"threads_started":1}
W20260619 20:55:32.410434 7601 tablet.cc:2369] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:32.531183 7465 tablet.cc:2369] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:55:32.539450 7737 tablet.cc:2369] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:32.568559 7845 raft_consensus.cc:482] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:32.568648 7845 raft_consensus.cc:504] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.568820 7845 leader_election.cc:288] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401), 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:32.569092 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dae5c20d5c4ea0a78cb991db88bfc8" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "246a46c735424910b13b74f0e1b1ccc9" is_pre_election: true
I20260619 20:55:32.569103 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dae5c20d5c4ea0a78cb991db88bfc8" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" is_pre_election: true
I20260619 20:55:32.569196 7688 raft_consensus.cc:2454] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 0.
I20260619 20:55:32.569196 7552 raft_consensus.cc:2454] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 0.
I20260619 20:55:32.569361 7352 leader_election.cc:302] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542d4c6b1c594a4292f1ac8dea2d2e2d, 92d75cfaaf874e508af1d249c620244f; no voters:
I20260619 20:55:32.569465 7845 raft_consensus.cc:2781] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:32.569558 7845 raft_consensus.cc:482] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:32.569586 7845 raft_consensus.cc:3037] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:32.570327 7845 raft_consensus.cc:504] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.570508 7845 leader_election.cc:288] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 election: Requested vote from peers 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401), 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421)
I20260619 20:55:32.570617 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dae5c20d5c4ea0a78cb991db88bfc8" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d"
I20260619 20:55:32.570657 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dae5c20d5c4ea0a78cb991db88bfc8" candidate_uuid: "92d75cfaaf874e508af1d249c620244f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "246a46c735424910b13b74f0e1b1ccc9"
I20260619 20:55:32.570696 7688 raft_consensus.cc:3037] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:32.570729 7552 raft_consensus.cc:3037] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:32.571441 7552 raft_consensus.cc:2454] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 1.
I20260619 20:55:32.571441 7688 raft_consensus.cc:2454] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d75cfaaf874e508af1d249c620244f in term 1.
I20260619 20:55:32.571621 7350 leader_election.cc:302] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246a46c735424910b13b74f0e1b1ccc9, 92d75cfaaf874e508af1d249c620244f; no voters:
I20260619 20:55:32.571715 7845 raft_consensus.cc:2781] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:32.571843 7845 raft_consensus.cc:686] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 1 LEADER]: Becoming Leader. State: Replica: 92d75cfaaf874e508af1d249c620244f, State: Running, Role: LEADER
I20260619 20:55:32.571885 7845 consensus_queue.cc:237] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } }
I20260619 20:55:32.572441 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.572149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:55243)
0619 20:55:32.572201 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.572405 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:55:32.572587 7280 catalog_manager.cc:5495] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d75cfaaf874e508af1d249c620244f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "92d75cfaaf874e508af1d249c620244f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d75cfaaf874e508af1d249c620244f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:32.605391 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.603052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.1:36767)
0619 20:55:32.603108 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:32.603110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:32.603119 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:32.603178 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:32.603179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:32.603191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:32.603249 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:32.603378 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.603946 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.604704 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:32.604910 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:32.604985 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:32.605124 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:32.605155 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:32.605230 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:32.605289 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:55:32.605308 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:55:32.606243 7688 raft_consensus.cc:1261] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Refusing update from remote peer 92d75cfaaf874e508af1d249c620244f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:32.606269 7552 raft_consensus.cc:1261] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Refusing update from remote peer 92d75cfaaf874e508af1d249c620244f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:32.606472 7845 consensus_queue.cc:1035] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Connected to new peer: Peer: permanent_uuid: "542d4c6b1c594a4292f1ac8dea2d2e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:32.606572 7850 consensus_queue.cc:1035] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [LEADER]: Connected to new peer: Peer: permanent_uuid: "246a46c735424910b13b74f0e1b1ccc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:32.607268 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.607154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:47915)
0619 20:55:32.607201 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.607230 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:55:32.607831 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.607721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:35309)
0619 20:55:32.607765 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.607799 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 20:55:32.609827 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.609642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.1:32929)
0619 20:55:32.609772 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.609799 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 20:55:32.609855 7855 mvcc.cc:205] Tried to move back new op lower bound from 7298672773553238016 to 7298672773415063552. Current Snapshot: MvccSnapshot[applied={T|T < 7298672773553238016}]
I20260619 20:55:32.609930 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.609832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.3:49171)
0619 20:55:32.609887 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.609908 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:55:32.610350 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.610198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45283 (local address 127.0.98.2:48557)
0619 20:55:32.610295 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.610320 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 20:55:32.611862 7279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:32.611934 7279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:32.613070 7279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72b9bff20d7c4541b03c8131b96507e5: Sending DeleteTablet for 3 replicas of tablet d8dae5c20d5c4ea0a78cb991db88bfc8
I20260619 20:55:32.613354 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43370
I20260619 20:55:32.613354 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41494
I20260619 20:55:32.613512 7668 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dae5c20d5c4ea0a78cb991db88bfc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:32 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43732
I20260619 20:55:32.613680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7333
I20260619 20:55:32.613576 7863 tablet_replica.cc:330] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f: stopping tablet replica
I20260619 20:55:32.613792 7863 raft_consensus.cc:2229] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:32.613875 7863 raft_consensus.cc:2258] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 92d75cfaaf874e508af1d249c620244f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:32.614265 7862 tablet_replica.cc:330] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: stopping tablet replica
I20260619 20:55:32.614307 7862 raft_consensus.cc:2229] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:32.614346 7862 raft_consensus.cc:2258] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:32.614444 7864 tablet_replica.cc:330] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: stopping tablet replica
I20260619 20:55:32.614490 7864 raft_consensus.cc:2229] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:32.614528 7864 raft_consensus.cc:2258] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:32.614720 7862 ts_tablet_manager.cc:1905] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:32.614750 7864 ts_tablet_manager.cc:1905] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:32.616179 7862 ts_tablet_manager.cc:1918] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:32.616179 7864 ts_tablet_manager.cc:1918] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:32.616240 7862 log.cc:1196] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/d8dae5c20d5c4ea0a78cb991db88bfc8
I20260619 20:55:32.616240 7864 log.cc:1196] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/d8dae5c20d5c4ea0a78cb991db88bfc8
I20260619 20:55:32.616466 7862 ts_tablet_manager.cc:1939] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 246a46c735424910b13b74f0e1b1ccc9: Deleting consensus metadata
I20260619 20:55:32.616466 7864 ts_tablet_manager.cc:1939] T d8dae5c20d5c4ea0a78cb991db88bfc8 P 542d4c6b1c594a4292f1ac8dea2d2e2d: Deleting consensus metadata
I20260619 20:55:32.616767 7266 catalog_manager.cc:4841] TS 246a46c735424910b13b74f0e1b1ccc9 (127.0.98.2:46421): tablet d8dae5c20d5c4ea0a78cb991db88bfc8 (table test-table [id=42335afa02c5473582857847da68cb39]) successfully deleted
I20260619 20:55:32.616829 7268 catalog_manager.cc:4841] TS 542d4c6b1c594a4292f1ac8dea2d2e2d (127.0.98.3:45401): tablet d8dae5c20d5c4ea0a78cb991db88bfc8 (table test-table [id=42335afa02c5473582857847da68cb39]) successfully deleted
I20260619 20:55:32.620373 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:32.620256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36767 (local address 127.0.0.1:43398)
0619 20:55:32.620307 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:32.620336 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:55:32.620510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7469
W20260619 20:55:32.620519 7265 catalog_manager.cc:4584] TS 92d75cfaaf874e508af1d249c620244f (127.0.98.1:36767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8dae5c20d5c4ea0a78cb991db88bfc8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36767: connect: Connection refused (error 111)
I20260619 20:55:32.626276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7605
I20260619 20:55:32.630538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 6954
I20260619 20:55:32.635017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7142
2026-06-19T20:55:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7190 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:32 dist-test-slave-8bgz krb5kdc[7867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:32 dist-test-slave-8bgz krb5kdc[7867](info): set up 2 sockets
Jun 19 20:55:32 dist-test-slave-8bgz krb5kdc[7867](info): commencing operation
krb5kdc: starting...
W20260619 20:55:33.726822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:33 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 20:55:33.837459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:33.916057 7880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:33.916241 7880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:33.916263 7880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:33.917347 7880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:33.917380 7880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:55:33.917394 7880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:33.917407 7880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:33.917419 7880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:33.917430 7880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:33.918816 7880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:33.918983 7880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:33.919132 7880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:33.923620 7885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:33.923700 7888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:33.923636 7886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:33.923653 7880 server_base.cc:1034] running on GCE node
I20260619 20:55:33.924116 7880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:33.924412 7880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:33.925581 7880 hybrid_clock.cc:648] HybridClock initialized: now 1781902533925560 us; error 34 us; skew 500 ppm
Jun 19 20:55:33 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902533, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:33.927960 7880 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 20:55:33.928371 7880 webserver.cc:460] Webserver started at http://127.0.98.62:42085/ using document root <none> and password file <none>
I20260619 20:55:33.928560 7880 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:33.928606 7880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:33.928690 7880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:33.929610 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "922e3fd996194f40806c6d95407c5ba3"
format_stamp: "Formatted at 2026-06-19 20:55:33 on dist-test-slave-8bgz"
server_key: "e64b9c14216b296a05452a0b01bcc2ab"
server_key_iv: "2fd1ebfbc48b592c79f425cdc1584c14"
server_key_version: "encryptionkey@0"
I20260619 20:55:33.929905 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "922e3fd996194f40806c6d95407c5ba3"
format_stamp: "Formatted at 2026-06-19 20:55:33 on dist-test-slave-8bgz"
server_key: "e64b9c14216b296a05452a0b01bcc2ab"
server_key_iv: "2fd1ebfbc48b592c79f425cdc1584c14"
server_key_version: "encryptionkey@0"
I20260619 20:55:33.931178 7880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:33.931820 7895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:33.931985 7880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:33.932058 7880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "922e3fd996194f40806c6d95407c5ba3"
format_stamp: "Formatted at 2026-06-19 20:55:33 on dist-test-slave-8bgz"
server_key: "e64b9c14216b296a05452a0b01bcc2ab"
server_key_iv: "2fd1ebfbc48b592c79f425cdc1584c14"
server_key_version: "encryptionkey@0"
I20260619 20:55:33.932111 7880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:33.946571 7880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:33.948238 7880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:33.948340 7880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:33.952579 7880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45977
I20260619 20:55:33.952651 7957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45977 every 8 connection(s)
I20260619 20:55:33.952934 7880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:33.953562 7958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:33.955708 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Bootstrap starting.
I20260619 20:55:33.956264 7958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:33.956544 7958 log.cc:824] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:33.957167 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: No bootstrap required, opened a new log
I20260619 20:55:33.959511 7958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } }
I20260619 20:55:33.959657 7958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:33.959700 7958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922e3fd996194f40806c6d95407c5ba3, State: Initialized, Role: FOLLOWER
I20260619 20:55:33.959833 7958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } }
I20260619 20:55:33.959903 7958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:33.959933 7958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:33.959981 7958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:33.960462 7958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } }
I20260619 20:55:33.960568 7958 leader_election.cc:302] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 922e3fd996194f40806c6d95407c5ba3; no voters:
I20260619 20:55:33.960719 7958 leader_election.cc:288] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:33.960762 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:33.960917 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 922e3fd996194f40806c6d95407c5ba3, State: Running, Role: LEADER
I20260619 20:55:33.960959 7958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:33.961027 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } }
I20260619 20:55:33.961376 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 922e3fd996194f40806c6d95407c5ba3. Latest consensus state: current_term: 1 leader_uuid: "922e3fd996194f40806c6d95407c5ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } } }
I20260619 20:55:33.961486 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:33.961371 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "922e3fd996194f40806c6d95407c5ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922e3fd996194f40806c6d95407c5ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45977 } } }
I20260619 20:55:33.961702 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:33.961741 7968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:33.961890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7880
I20260619 20:55:33.961995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:33.962186 392 external_mini_cluster.cc:1428] Setting key cc61b63e0b4103402f6f00212b96e881
I20260619 20:55:33.962317 7968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:33.963845 7968 catalog_manager.cc:1347] Generated new cluster ID: 6103560c128e4351bba582bead0b20b3
I20260619 20:55:33.963934 7968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:55:33 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:33.974342 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:33.962846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.0.98.62:45977)
0619 20:55:33.963012 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:33.963020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:33.963389 (+ 369us) server_negotiation.cc:407] Connection header received
0619 20:55:33.963635 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:33.963643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:33.963696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:33.963868 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:33.964614 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:33.965978 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:33.967053 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:33.968097 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:33.968197 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:33.970860 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:33.970864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:33.970864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:33.970875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:33.972819 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:33.973161 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:33.973164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:33.973164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:33.973266 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:33.973449 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:33.973450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:33.973451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:33.973744 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:33.973900 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:33.973963 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:55:33.974094 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260619 20:55:33.994323 7968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:33.994814 7968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:33.998641 7968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Generated new TSK 0
I20260619 20:55:33.998828 7968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 20:55:34.025123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45977
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:34.104943 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:34.105146 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:34.105168 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:34.106254 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:34.106289 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:34.106309 7984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:34.106328 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:34.107676 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:34.107832 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:34.107985 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:55:34.112872 7984 server_base.cc:1034] running on GCE node
W20260619 20:55:34.112844 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:34.112834 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:34.112854 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:34.113214 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:34.113444 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:34.114575 7984 hybrid_clock.cc:648] HybridClock initialized: now 1781902534114561 us; error 33 us; skew 500 ppm
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:34.116797 7984 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:55:34.117190 7984 webserver.cc:460] Webserver started at http://127.0.98.1:43173/ using document root <none> and password file <none>
I20260619 20:55:34.117398 7984 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:34.117458 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:34.117594 7984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:34.118634 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5585e0123f44618883d45fa602b1b3c"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "4ef8e90d758b5c4921976886c45d189c"
server_key_iv: "d2c3ae43f6c13984f05e6dc6045176a9"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.119012 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5585e0123f44618883d45fa602b1b3c"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "4ef8e90d758b5c4921976886c45d189c"
server_key_iv: "d2c3ae43f6c13984f05e6dc6045176a9"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.120220 7984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:34.120923 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.121098 7984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:34.121186 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "d5585e0123f44618883d45fa602b1b3c"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "4ef8e90d758b5c4921976886c45d189c"
server_key_iv: "d2c3ae43f6c13984f05e6dc6045176a9"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.121272 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:34.127962 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:34.129570 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:34.129720 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:34.130156 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:34.130234 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.130270 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:34.130301 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.136812 7984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37907
I20260619 20:55:34.136924 8114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37907 every 8 connection(s)
I20260619 20:55:34.137174 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:34.139350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 7984
I20260619 20:55:34.139462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:34.139655 392 external_mini_cluster.cc:1428] Setting key 64d2c3275fa176630bbd42acee7732b6
I20260619 20:55:34.140590 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.130604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.1:44231)
0619 20:55:34.130786 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.130824 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.131310 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.131596 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.131601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.131986 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.132282 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.132290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.133294 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.133296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.134765 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.134769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.135271 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.135272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.135425 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.135990 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.136000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.137714 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.139100 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.139106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.139107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.139322 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.139558 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.139559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.139559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.139655 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.140140 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.140145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.140255 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.140342 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.140451 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260619 20:55:34.140599 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.130668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44231 (local address 127.0.98.62:45977)
0619 20:55:34.130718 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.130720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.130913 (+ 193us) server_negotiation.cc:407] Connection header received
0619 20:55:34.131415 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.131416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.131432 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.131490 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.132399 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.133200 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.134858 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.135180 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.135270 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.137808 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.137810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.137810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.137819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.139034 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.139422 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.139424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.139424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.139486 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.139750 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.139752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.139752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.139913 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.139986 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.140372 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.140466 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:55:34.142015 8115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45977
I20260619 20:55:34.142096 8115 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:34.142303 8115 heartbeater.cc:511] Master 127.0.98.62:45977 requested a full tablet report, sending...
I20260619 20:55:34.142792 7911 ts_manager.cc:194] Registered new tserver with Master: d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907)
I20260619 20:55:34.143404 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44231
I20260619 20:55:34.148373 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.142274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.1:47631)
0619 20:55:34.142420 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.142430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.142500 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.142621 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.142622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.142786 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.142833 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.142834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.143794 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.143795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.144517 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.144519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.145011 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.145011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.145053 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.145460 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.145464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.145717 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.147435 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.147436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.147436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.147593 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.147808 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.147808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.147808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.147874 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.148162 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.148163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.148194 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.148242 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.148308 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":23,"thread_start_us":69,"threads_started":1}
I20260619 20:55:34.148447 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.142139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47631 (local address 127.0.98.62:45977)
0619 20:55:34.142194 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.142197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.142498 (+ 301us) server_negotiation.cc:407] Connection header received
0619 20:55:34.142550 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.142551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.142562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.142616 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.142909 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.143685 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.144595 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.144935 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.145016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.145812 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.145813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.145814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.145821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.147357 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.147661 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.147662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.147662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.147717 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.147966 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.147967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.147967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.148097 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.148175 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.148271 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.148359 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 20:55:34.165187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45977
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:34.243790 8124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:34.243983 8124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:34.244005 8124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:34.245070 8124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:34.245102 8124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:34.245121 8124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:34.245137 8124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:34.246558 8124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:34.246727 8124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:34.246884 8124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:34.251538 8132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:34.251657 8124 server_base.cc:1034] running on GCE node
W20260619 20:55:34.251519 8130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:34.251518 8129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:34.251948 8124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:34.252177 8124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:34.253328 8124 hybrid_clock.cc:648] HybridClock initialized: now 1781902534253308 us; error 50 us; skew 500 ppm
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:34.255738 8124 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:55:34.256167 8124 webserver.cc:460] Webserver started at http://127.0.98.2:41483/ using document root <none> and password file <none>
I20260619 20:55:34.256381 8124 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:34.256443 8124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:34.256587 8124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:34.257550 8124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "8e1a27dabd670cec04b88fdd7cb3a4ad"
server_key_iv: "da0ca1e747d10ddee8fb9f8c42af660c"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.257897 8124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "8e1a27dabd670cec04b88fdd7cb3a4ad"
server_key_iv: "da0ca1e747d10ddee8fb9f8c42af660c"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.259249 8124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:55:34.259903 8139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.260054 8124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.260149 8124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "8e1a27dabd670cec04b88fdd7cb3a4ad"
server_key_iv: "da0ca1e747d10ddee8fb9f8c42af660c"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.260216 8124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:34.278031 8124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:34.280278 8124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:34.280438 8124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:34.280966 8124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:34.281019 8124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.281052 8124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:34.281080 8124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:34.288367 8124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37127
I20260619 20:55:34.288436 8254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37127 every 8 connection(s)
I20260619 20:55:34.288725 8124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:34.289579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8124
I20260619 20:55:34.289690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:34.289870 392 external_mini_cluster.cc:1428] Setting key a4300df0974d26c62e92a5f756998e87
I20260619 20:55:34.291690 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.281319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.2:46349)
0619 20:55:34.281515 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.281565 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.282280 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.282554 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.282560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.282957 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.283251 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.283258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.284344 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.284345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.285797 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.285800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.286236 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.286237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.286383 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.286830 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.286840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.288580 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.290423 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.290428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.290430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.290692 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.290893 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.290893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.290893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.290971 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.291237 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.291241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.291368 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.291445 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.291551 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260619 20:55:34.291705 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.281320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46349 (local address 127.0.98.62:45977)
0619 20:55:34.281385 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.281388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.281668 (+ 280us) server_negotiation.cc:407] Connection header received
0619 20:55:34.282383 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.282383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.282393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.282466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.283379 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.284217 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.285896 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.286122 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.286175 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.288679 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.288680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.288680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.288688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.290356 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.290771 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.290772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.290773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.290832 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.291022 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.291023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.291024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.291166 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.291231 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.291486 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.291572 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:55:34.292279 8255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45977
I20260619 20:55:34.292363 8255 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:34.292528 8255 heartbeater.cc:511] Master 127.0.98.62:45977 requested a full tablet report, sending...
I20260619 20:55:34.293437 7911 ts_manager.cc:194] Registered new tserver with Master: 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127)
I20260619 20:55:34.293812 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46349
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 20:55:34.303194 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.293370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.2:38885)
0619 20:55:34.297957 (+ 4587us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.297967 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.298019 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.298147 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.298148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.298365 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.298417 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.298418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.299046 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.299047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.299544 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.299546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.299831 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.299831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.299887 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.300255 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.300263 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.300582 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.302171 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.302172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.302172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.302376 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.302581 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.302581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.302581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.302628 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.302983 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.302983 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.303014 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.303054 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.303123 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4565,"thread_start_us":633,"threads_started":1}
I20260619 20:55:34.303296 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.293433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38885 (local address 127.0.98.62:45977)
0619 20:55:34.293468 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.293471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.298016 (+ 4545us) server_negotiation.cc:407] Connection header received
0619 20:55:34.298062 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.298063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.298076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.298146 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.298482 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.298980 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.299607 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.299780 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.299825 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.300654 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.300655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.300656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.300663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.302082 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.302441 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.302442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.302443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.302516 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.302695 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.302696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.302696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.302915 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.302980 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.303090 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.303168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:55:34.314967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45977
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:34.396207 8264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:34.396392 8264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:34.396414 8264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:34.397544 8264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:34.397585 8264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:34.397629 8264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:55:34.397666 8264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:34.399135 8264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:34.399362 8264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:34.399583 8264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:34.404357 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:34.404338 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:34.404426 8264 server_base.cc:1034] running on GCE node
W20260619 20:55:34.404337 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:34.404680 8264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:34.404861 8264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:34.405987 8264 hybrid_clock.cc:648] HybridClock initialized: now 1781902534405977 us; error 26 us; skew 500 ppm
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:34.408285 8264 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:55:34.408690 8264 webserver.cc:460] Webserver started at http://127.0.98.3:36033/ using document root <none> and password file <none>
I20260619 20:55:34.408919 8264 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:34.408982 8264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:34.409119 8264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:34.410089 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9067a5330ad4bfb814116f1c0e12793"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "fffff0a0eccf8a0770579c28e33257be"
server_key_iv: "450a9349878df326e61367232ea7f436"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.410492 8264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9067a5330ad4bfb814116f1c0e12793"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "fffff0a0eccf8a0770579c28e33257be"
server_key_iv: "450a9349878df326e61367232ea7f436"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.411722 8264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:34.412362 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.412525 8264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:34.412607 8264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e9067a5330ad4bfb814116f1c0e12793"
format_stamp: "Formatted at 2026-06-19 20:55:34 on dist-test-slave-8bgz"
server_key: "fffff0a0eccf8a0770579c28e33257be"
server_key_iv: "450a9349878df326e61367232ea7f436"
server_key_version: "encryptionkey@0"
I20260619 20:55:34.412683 8264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:34.427193 8264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:34.428830 8264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:34.429000 8264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:34.429476 8264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:34.429535 8264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.429574 8264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:34.429605 8264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.436038 8264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36789
I20260619 20:55:34.436108 8394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36789 every 8 connection(s)
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:34.436416 8264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:34.439149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8264
I20260619 20:55:34.439246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:34.439474 392 external_mini_cluster.cc:1428] Setting key d5d5da8ac6e5a02d5a7db602c9187d94
I20260619 20:55:34.439584 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.429940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42169 (local address 127.0.98.62:45977)
0619 20:55:34.430003 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.430005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.430192 (+ 187us) server_negotiation.cc:407] Connection header received
0619 20:55:34.430694 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.430695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.430707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.430796 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.431605 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.432417 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.433965 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.434316 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.434412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.436896 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.436897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.436898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.436905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.438330 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.438676 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.438676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.438677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.438724 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.438902 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.438903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.438903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.439040 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.439118 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.439397 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.439479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:34.439708 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.429924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.3:42169)
0619 20:55:34.430067 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.430103 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.430601 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.430877 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.430880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.431224 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.431504 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.431511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.432510 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.432511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.433871 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.433874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.434411 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.434412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.434506 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.435025 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.435035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.436788 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.438407 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.438411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.438412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.438615 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.438779 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.438779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.438779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.438857 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.439109 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.439113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.439251 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.439352 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.439473 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260619 20:55:34.440457 8395 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45977
I20260619 20:55:34.440541 8395 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:34.440698 8395 heartbeater.cc:511] Master 127.0.98.62:45977 requested a full tablet report, sending...
I20260619 20:55:34.441083 7910 ts_manager.cc:194] Registered new tserver with Master: e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.441514 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42169
I20260619 20:55:34.446393 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.441296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56651 (local address 127.0.98.62:45977)
0619 20:55:34.441333 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.441336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.441428 (+ 92us) server_negotiation.cc:407] Connection header received
0619 20:55:34.441486 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.441486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.441496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.441552 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.441922 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.442483 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.443146 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.443344 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.443421 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.444247 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.444249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.444249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.444262 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.445328 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.445648 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.445649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.445650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.445705 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.445903 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.445903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.445904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.446004 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.446072 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.446239 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.446310 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:34.446405 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.441255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.98.3:56651)
0619 20:55:34.441369 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.441378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.441428 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.441555 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.441555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.441753 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.441825 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.441827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.442571 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.442571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.443069 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.443071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.443423 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.443424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.443486 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.443883 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.443888 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.444147 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.445406 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.445407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.445407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.445568 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.445773 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.445773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.445774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.445828 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.446065 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.446066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.446105 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.446188 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.446302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260619 20:55:34.451085 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:55:34.471508 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:34.466429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.0.98.62:45977)
0619 20:55:34.466483 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.466485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.466499 (+ 14us) server_negotiation.cc:407] Connection header received
0619 20:55:34.466578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.466579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.466588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.466644 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.467213 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.467690 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.468395 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.468606 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.468677 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.471419 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43860: BlockingRecv error: recv got EOF from 127.0.0.1:43860 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:34.478253 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.472173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.62:45977)
0619 20:55:34.472227 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.472229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.472305 (+ 76us) server_negotiation.cc:407] Connection header received
0619 20:55:34.472368 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.472369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.472378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.472425 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.472851 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.473300 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.473965 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.474140 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.474205 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.476096 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.476097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.476098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.476105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.477181 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.477501 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.477502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.477503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.477566 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.477752 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.477753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.477754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.477857 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.477952 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.478053 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.478127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:55:34.479544 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:34.481000 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:34.487355 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.482998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36789 (local address 127.0.0.1:52858)
0619 20:55:34.483377 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.483387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.483435 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.483893 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.483894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.483896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.483942 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.483943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.484824 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.484825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.486133 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.486136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.487060 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.487135 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.487149 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.487225 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.487283 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":37,"thread_start_us":47,"threads_started":1}
I20260619 20:55:34.487527 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.483071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.0.98.2:37127)
0619 20:55:34.483425 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.483430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.483442 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:55:34.483485 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.483487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.483503 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.483580 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.483777 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.484903 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.486264 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.486985 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.487140 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.487395 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.487440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":87,"threads_started":1}
I20260619 20:55:34.487414 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.483206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.98.3:36789)
0619 20:55:34.483694 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.483698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.483708 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:34.483740 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.483742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.483755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.483827 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.484017 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.484733 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.486257 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.486985 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.487137 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.487256 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.487316 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":96,"threads_started":1}
I20260619 20:55:34.487682 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.482981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.0.1:60934)
0619 20:55:34.483248 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.483293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.483358 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.483592 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.483615 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.483618 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.483682 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.483683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.484995 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.484995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.486133 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.486136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.487167 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.487249 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.487252 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.487304 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.487342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260619 20:55:34.488233 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.483009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37907 (local address 127.0.0.1:35822)
0619 20:55:34.483457 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.483466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.483513 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.483749 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.483750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.483751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.483791 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.483792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.486256 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.486257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.486909 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.486910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.488022 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.488023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.488084 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.488088 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.488138 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.488172 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":43,"threads_started":1}
I20260619 20:55:34.488268 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 6c691c1b39b04b28be0f982a327e653c (DEFAULT_TABLE table=test-table [id=1492e291ba67473bb625f053d3ff5cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.488435 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 6c691c1b39b04b28be0f982a327e653c (DEFAULT_TABLE table=test-table [id=1492e291ba67473bb625f053d3ff5cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.488554 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.483215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.0.98.1:37907)
0619 20:55:34.483365 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.483371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.483520 (+ 149us) server_negotiation.cc:407] Connection header received
0619 20:55:34.483590 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.483593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.483608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.483682 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.483868 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.484911 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487005 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.487835 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.487901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.487972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.488418 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.488465 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260619 20:55:34.488677 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c691c1b39b04b28be0f982a327e653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.488778 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c691c1b39b04b28be0f982a327e653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.489122 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 6c691c1b39b04b28be0f982a327e653c (DEFAULT_TABLE table=test-table [id=1492e291ba67473bb625f053d3ff5cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.489410 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c691c1b39b04b28be0f982a327e653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.491843 8420 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Bootstrap starting.
I20260619 20:55:34.492362 8421 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Bootstrap starting.
I20260619 20:55:34.492391 8420 tablet_bootstrap.cc:654] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.492717 8420 log.cc:824] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:34.493021 8421 tablet_bootstrap.cc:654] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.493355 8421 log.cc:824] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:34.493479 8420 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: No bootstrap required, opened a new log
I20260619 20:55:34.493543 8420 ts_tablet_manager.cc:1397] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:34.493870 8423 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Bootstrap starting.
I20260619 20:55:34.494089 8421 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: No bootstrap required, opened a new log
I20260619 20:55:34.494155 8421 ts_tablet_manager.cc:1397] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:34.494415 8423 tablet_bootstrap.cc:654] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.494683 8423 log.cc:824] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:34.495275 8423 tablet_bootstrap.cc:492] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: No bootstrap required, opened a new log
I20260619 20:55:34.495347 8423 ts_tablet_manager.cc:1397] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:55:34.495507 8420 raft_consensus.cc:348] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.495659 8420 raft_consensus.cc:374] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.495723 8420 raft_consensus.cc:729] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.495913 8420 consensus_queue.cc:260] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.496057 8421 raft_consensus.cc:348] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.496163 8420 ts_tablet_manager.cc:1428] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:55:34.496245 8421 raft_consensus.cc:374] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.496289 8255 heartbeater.cc:503] Master 127.0.98.62:45977 was elected leader, sending a full tablet report...
I20260619 20:55:34.496310 8421 raft_consensus.cc:729] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.496438 8421 consensus_queue.cc:260] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.496690 8421 ts_tablet_manager.cc:1428] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:34.496692 8395 heartbeater.cc:503] Master 127.0.98.62:45977 was elected leader, sending a full tablet report...
I20260619 20:55:34.497257 8423 raft_consensus.cc:348] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.497388 8423 raft_consensus.cc:374] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.497448 8423 raft_consensus.cc:729] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.497548 8423 consensus_queue.cc:260] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.497776 8423 ts_tablet_manager.cc:1428] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:55:34.497839 8115 heartbeater.cc:503] Master 127.0.98.62:45977 was elected leader, sending a full tablet report...
W20260619 20:55:34.539474 8256 tablet.cc:2369] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:34.599040 8428 raft_consensus.cc:482] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:34.599151 8428 raft_consensus.cc:504] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.599445 8428 leader_election.cc:288] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.602479 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.599844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40971 (local address 127.0.98.3:36789)
0619 20:55:34.599910 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.599912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.599921 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:34.599948 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.599950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.599964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.600038 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.600205 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.600724 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.601681 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.602059 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.602137 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.602147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.602368 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.602406 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:55:34.602490 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.599849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60513 (local address 127.0.98.2:37127)
0619 20:55:34.599894 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.599896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.599912 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:55:34.599964 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.599965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.599974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.600053 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.600251 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.600677 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.601678 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.602057 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.602144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.602155 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.602375 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.602412 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:55:34.602524 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.599644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36789 (local address 127.0.98.1:40971)
0619 20:55:34.599707 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.599720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.599766 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.600043 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.600044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.600046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.600125 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.600127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.600827 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.600828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.601593 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.601595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.602147 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.602149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.602287 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.602290 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.602360 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.602402 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":20}
I20260619 20:55:34.602552 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.599644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.98.1:60513)
0619 20:55:34.599830 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.599839 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.599899 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.600070 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.600071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.600073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.600150 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.600151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.600755 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.600756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.601579 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.601582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.602145 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.602146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.602281 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.602285 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.602355 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.602401 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":26,"thread_start_us":101,"threads_started":1}
I20260619 20:55:34.602953 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c691c1b39b04b28be0f982a327e653c" candidate_uuid: "d5585e0123f44618883d45fa602b1b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793" is_pre_election: true
I20260619 20:55:34.602978 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c691c1b39b04b28be0f982a327e653c" candidate_uuid: "d5585e0123f44618883d45fa602b1b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" is_pre_election: true
I20260619 20:55:34.603111 8349 raft_consensus.cc:2454] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5585e0123f44618883d45fa602b1b3c in term 0.
I20260619 20:55:34.603127 8209 raft_consensus.cc:2454] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5585e0123f44618883d45fa602b1b3c in term 0.
I20260619 20:55:34.603338 8002 leader_election.cc:302] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5585e0123f44618883d45fa602b1b3c, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:34.603456 8428 raft_consensus.cc:2781] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:34.603516 8428 raft_consensus.cc:482] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:34.603533 8428 raft_consensus.cc:3037] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.604429 8428 raft_consensus.cc:504] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.604558 8428 leader_election.cc:288] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [CANDIDATE]: Term 1 election: Requested vote from peers 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.604717 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c691c1b39b04b28be0f982a327e653c" candidate_uuid: "d5585e0123f44618883d45fa602b1b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
I20260619 20:55:34.604717 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c691c1b39b04b28be0f982a327e653c" candidate_uuid: "d5585e0123f44618883d45fa602b1b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793"
I20260619 20:55:34.604792 8349 raft_consensus.cc:3037] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.604792 8209 raft_consensus.cc:3037] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.605588 8209 raft_consensus.cc:2454] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5585e0123f44618883d45fa602b1b3c in term 1.
I20260619 20:55:34.605588 8349 raft_consensus.cc:2454] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5585e0123f44618883d45fa602b1b3c in term 1.
I20260619 20:55:34.605753 8002 leader_election.cc:302] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5585e0123f44618883d45fa602b1b3c, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:34.605859 8428 raft_consensus.cc:2781] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:34.606010 8428 raft_consensus.cc:686] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 1 LEADER]: Becoming Leader. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Running, Role: LEADER
I20260619 20:55:34.606106 8428 consensus_queue.cc:237] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } }
I20260619 20:55:34.606901 7912 catalog_manager.cc:5495] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c reported cstate change: term changed from 0 to 1, leader changed from <none> to d5585e0123f44618883d45fa602b1b3c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5585e0123f44618883d45fa602b1b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } health_report { overall_health: HEALTHY } } }
I20260619 20:55:34.631232 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.629047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.0.98.1:37907)
0619 20:55:34.629103 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.629104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.629114 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:34.629222 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.629222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.629239 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.629294 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:34.629425 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.629866 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.630561 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.630733 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.630788 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.630906 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.631031 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.631090 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.631121 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.631136 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:55:34.632906 8209 raft_consensus.cc:1261] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Refusing update from remote peer d5585e0123f44618883d45fa602b1b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.632906 8349 raft_consensus.cc:1261] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Refusing update from remote peer d5585e0123f44618883d45fa602b1b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.633205 8428 consensus_queue.cc:1035] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.633329 8430 consensus_queue.cc:1035] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.636313 8434 mvcc.cc:205] Tried to move back new op lower bound from 7298672781853474816 to 7298672781747077120. Current Snapshot: MvccSnapshot[applied={T|T < 7298672781853474816}]
I20260619 20:55:34.637655 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:34.637745 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:34.638446 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Sending DeleteTablet for 3 replicas of tablet 6c691c1b39b04b28be0f982a327e653c
I20260619 20:55:34.638753 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c691c1b39b04b28be0f982a327e653c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260619 20:55:34.638867 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c691c1b39b04b28be0f982a327e653c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60934
I20260619 20:55:34.638947 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c691c1b39b04b28be0f982a327e653c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 20:55:34.639101 8444 tablet_replica.cc:330] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: stopping tablet replica
I20260619 20:55:34.639146 8443 tablet_replica.cc:330] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: stopping tablet replica
I20260619 20:55:34.639231 8444 raft_consensus.cc:2229] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:34.639261 8443 raft_consensus.cc:2229] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:34.639308 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:55:34.639389 8444 raft_consensus.cc:2258] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:34.639400 8443 raft_consensus.cc:2258] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:55:34.639432 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:34.639814 8443 ts_tablet_manager.cc:1905] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:34.639814 8444 ts_tablet_manager.cc:1905] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:34.640898 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 208e7034e6e14966b3ccab615337f48d (DEFAULT_TABLE table=test-table [id=85aa78e90a4746eaa2023a2e54a1244e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.640899 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 208e7034e6e14966b3ccab615337f48d (DEFAULT_TABLE table=test-table [id=85aa78e90a4746eaa2023a2e54a1244e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.640995 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208e7034e6e14966b3ccab615337f48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.640995 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208e7034e6e14966b3ccab615337f48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.641095 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 208e7034e6e14966b3ccab615337f48d (DEFAULT_TABLE table=test-table [id=85aa78e90a4746eaa2023a2e54a1244e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:34.641191 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208e7034e6e14966b3ccab615337f48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.642297 8421 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Bootstrap starting.
I20260619 20:55:34.642354 8420 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Bootstrap starting.
I20260619 20:55:34.642432 8423 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Bootstrap starting.
I20260619 20:55:34.642835 8420 tablet_bootstrap.cc:654] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.642880 8445 tablet_replica.cc:330] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: stopping tablet replica
I20260619 20:55:34.642890 8423 tablet_bootstrap.cc:654] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.642966 8445 raft_consensus.cc:2229] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:34.643045 8421 tablet_bootstrap.cc:654] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.643072 8445 raft_consensus.cc:2258] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:34.643582 8445 ts_tablet_manager.cc:1905] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:34.644198 8443 ts_tablet_manager.cc:1918] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:34.644344 8443 log.cc:1196] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/6c691c1b39b04b28be0f982a327e653c
I20260619 20:55:34.644345 8444 ts_tablet_manager.cc:1918] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:34.644397 8444 log.cc:1196] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/6c691c1b39b04b28be0f982a327e653c
I20260619 20:55:34.644665 8444 ts_tablet_manager.cc:1939] T 6c691c1b39b04b28be0f982a327e653c P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting consensus metadata
I20260619 20:55:34.644665 8443 ts_tablet_manager.cc:1939] T 6c691c1b39b04b28be0f982a327e653c P d5585e0123f44618883d45fa602b1b3c: Deleting consensus metadata
I20260619 20:55:34.645032 7896 catalog_manager.cc:4841] TS d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907): tablet 6c691c1b39b04b28be0f982a327e653c (table test-table [id=1492e291ba67473bb625f053d3ff5cee]) successfully deleted
I20260619 20:55:34.645129 7899 catalog_manager.cc:4841] TS 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127): tablet 6c691c1b39b04b28be0f982a327e653c (table test-table [id=1492e291ba67473bb625f053d3ff5cee]) successfully deleted
I20260619 20:55:34.645282 8445 ts_tablet_manager.cc:1918] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:34.645355 8445 log.cc:1196] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/6c691c1b39b04b28be0f982a327e653c
I20260619 20:55:34.645506 8420 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: No bootstrap required, opened a new log
I20260619 20:55:34.645566 8420 ts_tablet_manager.cc:1397] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:55:34.645635 8445 ts_tablet_manager.cc:1939] T 6c691c1b39b04b28be0f982a327e653c P e9067a5330ad4bfb814116f1c0e12793: Deleting consensus metadata
I20260619 20:55:34.645713 8420 raft_consensus.cc:348] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.645797 8420 raft_consensus.cc:374] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.645823 8420 raft_consensus.cc:729] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.645865 8420 consensus_queue.cc:260] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.645960 8420 ts_tablet_manager.cc:1428] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:34.646008 7898 catalog_manager.cc:4841] TS e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789): tablet 6c691c1b39b04b28be0f982a327e653c (table test-table [id=1492e291ba67473bb625f053d3ff5cee]) successfully deleted
I20260619 20:55:34.646201 8421 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: No bootstrap required, opened a new log
I20260619 20:55:34.646266 8421 ts_tablet_manager.cc:1397] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:55:34.646418 8421 raft_consensus.cc:348] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.646477 8421 raft_consensus.cc:374] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.646497 8421 raft_consensus.cc:729] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.646540 8421 consensus_queue.cc:260] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.646608 8421 ts_tablet_manager.cc:1428] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.650542 8423 tablet_bootstrap.cc:492] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: No bootstrap required, opened a new log
I20260619 20:55:34.650599 8423 ts_tablet_manager.cc:1397] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260619 20:55:34.650754 8423 raft_consensus.cc:348] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.650841 8423 raft_consensus.cc:374] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.650874 8423 raft_consensus.cc:729] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.650919 8423 consensus_queue.cc:260] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.651021 8423 ts_tablet_manager.cc:1428] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 20:55:34.686996 8396 tablet.cc:2369] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:34.736927 8426 raft_consensus.cc:482] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:34.737038 8426 raft_consensus.cc:504] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.737353 8426 leader_election.cc:288] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.740887 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.737531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37907 (local address 127.0.98.2:53571)
0619 20:55:34.737580 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.737592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.737634 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.737836 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.737837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.737839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.737918 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.737920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.738654 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.738655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.739798 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.739800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.740629 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.740630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.740718 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.740722 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.740775 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.740816 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:55:34.740886 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.737639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53571 (local address 127.0.98.1:37907)
0619 20:55:34.737681 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.737684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.737692 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:34.737741 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.737742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.737751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.737832 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.737981 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.738571 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.739903 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.740512 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.740628 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.740640 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.740783 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.740819 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:34.741021 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.737615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60993 (local address 127.0.98.3:36789)
0619 20:55:34.737660 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.737662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.737851 (+ 189us) server_negotiation.cc:407] Connection header received
0619 20:55:34.737949 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.737951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.737963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.738007 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:34.738141 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.738883 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.740114 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.740603 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.740667 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.740677 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.740914 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.740961 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:55:34.741139 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208e7034e6e14966b3ccab615337f48d" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c" is_pre_election: true
I20260619 20:55:34.741225 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.737531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36789 (local address 127.0.98.2:60993)
0619 20:55:34.737798 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.737807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.737874 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.738016 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.738017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.738018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:34.738084 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.738085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.738971 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.738972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.740000 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.740002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.740729 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.740731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.740819 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.740824 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.741094 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.741135 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":142,"thread_start_us":80,"threads_started":1}
I20260619 20:55:34.741279 8069 raft_consensus.cc:2454] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 0.
I20260619 20:55:34.741447 8140 leader_election.cc:302] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, d5585e0123f44618883d45fa602b1b3c; no voters:
I20260619 20:55:34.741565 8426 raft_consensus.cc:2781] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:34.741601 8426 raft_consensus.cc:482] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:34.741621 8426 raft_consensus.cc:3037] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.741569 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208e7034e6e14966b3ccab615337f48d" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793" is_pre_election: true
I20260619 20:55:34.741649 8349 raft_consensus.cc:2454] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 0.
I20260619 20:55:34.742460 8426 raft_consensus.cc:504] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.742602 8426 leader_election.cc:288] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 election: Requested vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.742735 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208e7034e6e14966b3ccab615337f48d" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c"
I20260619 20:55:34.742766 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208e7034e6e14966b3ccab615337f48d" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793"
I20260619 20:55:34.742825 8069 raft_consensus.cc:3037] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.742853 8349 raft_consensus.cc:3037] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.743409 8069 raft_consensus.cc:2454] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 1.
I20260619 20:55:34.743592 8140 leader_election.cc:302] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, d5585e0123f44618883d45fa602b1b3c; no voters:
I20260619 20:55:34.743696 8426 raft_consensus.cc:2781] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:34.743799 8349 raft_consensus.cc:2454] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 1.
I20260619 20:55:34.743821 8426 raft_consensus.cc:686] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 LEADER]: Becoming Leader. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Running, Role: LEADER
I20260619 20:55:34.743883 8426 consensus_queue.cc:237] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.744522 7911 catalog_manager.cc:5495] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:55:34 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902533, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:34.777339 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.770809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.98.62:45977)
0619 20:55:34.770884 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.770886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.770998 (+ 112us) server_negotiation.cc:407] Connection header received
0619 20:55:34.771073 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.771074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.771084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.771149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:34.771663 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.772152 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.772929 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.773099 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.773214 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.775086 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:34.775088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:34.775088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:34.775095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:34.776231 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.776587 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.776588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.776589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.776640 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:34.776813 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:34.776814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:34.776814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:34.776916 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:34.776985 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.777127 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.777228 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:55:34.777418 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.770756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45977 (local address 127.0.0.1:43878)
0619 20:55:34.770925 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.770936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.771016 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.771152 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.771153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.771545 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:34.771587 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.771590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.772234 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.772235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.772843 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.772846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.773215 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.773216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.773269 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.773720 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:34.773724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:34.774994 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:34.776322 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.776329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.776331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.776509 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.776697 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:34.776697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:34.776698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:34.776753 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:34.776976 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:34.776982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:34.777028 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.777106 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.777287 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":86,"thread_start_us":98,"threads_started":1}
I20260619 20:55:34.779248 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:55:34.779419 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:34.780766 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 533e307cc4934c1583cebb3f936becb0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=afa839105f014e75b3fe6a67fe659f50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:34.780766 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 533e307cc4934c1583cebb3f936becb0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=afa839105f014e75b3fe6a67fe659f50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:34.780931 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e307cc4934c1583cebb3f936becb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.780934 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e307cc4934c1583cebb3f936becb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.780918 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 533e307cc4934c1583cebb3f936becb0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=afa839105f014e75b3fe6a67fe659f50]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:55:34.781032 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e307cc4934c1583cebb3f936becb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:34.781978 8423 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: Bootstrap starting.
I20260619 20:55:34.782322 8423 tablet_bootstrap.cc:654] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.782470 8420 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Bootstrap starting.
I20260619 20:55:34.782470 8421 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: Bootstrap starting.
I20260619 20:55:34.782845 8423 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: No bootstrap required, opened a new log
I20260619 20:55:34.782861 8420 tablet_bootstrap.cc:654] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.782861 8421 tablet_bootstrap.cc:654] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:34.782907 8423 ts_tablet_manager.cc:1397] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:34.783030 8423 raft_consensus.cc:348] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.783080 8423 raft_consensus.cc:374] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.783093 8423 raft_consensus.cc:729] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.783141 8423 consensus_queue.cc:260] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.783233 8423 ts_tablet_manager.cc:1428] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:34.783262 8440 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.783339 8440 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.783685 8421 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: No bootstrap required, opened a new log
I20260619 20:55:34.783737 8421 ts_tablet_manager.cc:1397] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:34.783813 8420 tablet_bootstrap.cc:492] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: No bootstrap required, opened a new log
I20260619 20:55:34.783864 8420 ts_tablet_manager.cc:1397] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:34.783900 8421 raft_consensus.cc:348] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.783964 8421 raft_consensus.cc:374] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.783993 8421 raft_consensus.cc:729] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.784003 8420 raft_consensus.cc:348] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.784042 8421 consensus_queue.cc:260] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.784070 8420 raft_consensus.cc:374] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:34.784093 8420 raft_consensus.cc:729] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Initialized, Role: FOLLOWER
I20260619 20:55:34.784137 8421 ts_tablet_manager.cc:1428] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:34.784132 8420 consensus_queue.cc:260] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.784263 8427 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.784358 8427 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.784504 8420 ts_tablet_manager.cc:1428] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:55:34.784528 8426 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.784607 8426 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 20:55:34.888051 8116 tablet.cc:2369] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:34.948678 8426 raft_consensus.cc:482] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:34.948777 8426 raft_consensus.cc:504] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.949008 8426 leader_election.cc:288] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.949235 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e307cc4934c1583cebb3f936becb0" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c" is_pre_election: true
I20260619 20:55:34.949235 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e307cc4934c1583cebb3f936becb0" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793" is_pre_election: true
I20260619 20:55:34.949353 8349 raft_consensus.cc:2454] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 0.
I20260619 20:55:34.949353 8069 raft_consensus.cc:2454] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 0.
I20260619 20:55:34.949507 8142 leader_election.cc:302] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:34.949627 8426 raft_consensus.cc:2781] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:34.949661 8426 raft_consensus.cc:482] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:34.949680 8426 raft_consensus.cc:3037] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.950310 8426 raft_consensus.cc:504] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.950436 8426 leader_election.cc:288] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 election: Requested vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789)
I20260619 20:55:34.950582 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e307cc4934c1583cebb3f936becb0" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c"
I20260619 20:55:34.950654 8069 raft_consensus.cc:3037] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.950615 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e307cc4934c1583cebb3f936becb0" candidate_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9067a5330ad4bfb814116f1c0e12793"
I20260619 20:55:34.950726 8349 raft_consensus.cc:3037] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:34.951287 8069 raft_consensus.cc:2454] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 1.
I20260619 20:55:34.951282 8349 raft_consensus.cc:2454] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e9a2a3fd6a4c8b8ef1149f57482b51 in term 1.
I20260619 20:55:34.951464 8140 leader_election.cc:302] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, d5585e0123f44618883d45fa602b1b3c, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:34.951570 8426 raft_consensus.cc:2781] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:34.951633 8426 raft_consensus.cc:686] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 LEADER]: Becoming Leader. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Running, Role: LEADER
I20260619 20:55:34.951681 8426 consensus_queue.cc:237] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:34.951972 8450 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: TxnStatusTablet state changed. Reason: New leader 83e9a2a3fd6a4c8b8ef1149f57482b51. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.952059 8450 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:34.952240 8462 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:55:34.952363 8462 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:55:34.952284 7912 catalog_manager.cc:5495] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: UNKNOWN } } }
I20260619 20:55:34.964915 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.962147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.0.98.2:37127)
0619 20:55:34.962184 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.962186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.962265 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:55:34.962366 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.962367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.962381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.962431 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:34.962551 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.963043 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.964216 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.964429 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.964481 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.964652 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.964733 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.964788 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.964853 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.964869 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:55:34.964972 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.962110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.0.1:60936)
0619 20:55:34.962221 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.962232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.962283 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.962444 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.962444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.962446 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:34.962495 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.962497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.963119 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.963120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.964131 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.964135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.964496 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.964497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.964558 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.964574 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.964789 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.964796 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.964851 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.964894 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 20:55:34.965960 8069 raft_consensus.cc:1261] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Refusing update from remote peer 83e9a2a3fd6a4c8b8ef1149f57482b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.965961 8349 raft_consensus.cc:1261] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Refusing update from remote peer 83e9a2a3fd6a4c8b8ef1149f57482b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.966198 8426 consensus_queue.cc:1035] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.966300 8450 consensus_queue.cc:1035] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.966809 8427 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: TxnStatusTablet state changed. Reason: New leader 83e9a2a3fd6a4c8b8ef1149f57482b51. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.966899 8427 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.966966 8440 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: TxnStatusTablet state changed. Reason: New leader 83e9a2a3fd6a4c8b8ef1149f57482b51. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.967062 8440 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.967418 8426 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.967490 8426 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:34.967617 8440 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.967682 8440 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P d5585e0123f44618883d45fa602b1b3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.968325 8426 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.968405 8426 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:55:34.968680 8435 mvcc.cc:205] Tried to move back new op lower bound from 7298672783218462720 to 7298672783162511360. Current Snapshot: MvccSnapshot[applied={T|T < 7298672783218462720}]
I20260619 20:55:34.968829 8427 tablet_replica.cc:439] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } }
I20260619 20:55:34.968905 8427 tablet_replica.cc:442] T 533e307cc4934c1583cebb3f936becb0 P e9067a5330ad4bfb814116f1c0e12793: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:55:34.971488 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.969024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.98.2:37127)
0619 20:55:34.969061 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.969063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.969069 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:34.969088 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.969089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.969097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.969141 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:34.969320 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.969828 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.970932 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.971122 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.971163 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.971307 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.971329 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.971373 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.971436 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.971448 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:34.983924 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.973727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41609 (local address 127.0.98.2:37127)
0619 20:55:34.973758 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:34.973759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:34.981125 (+ 7366us) server_negotiation.cc:407] Connection header received
0619 20:55:34.981151 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:34.981151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:34.981161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:34.981213 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:34.981354 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.982041 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.983196 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.983498 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.983567 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.983697 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.983730 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.983780 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:34.983824 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:55:34.983839 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 20:55:34.983939 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:34.973679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.98.2:41609)
0619 20:55:34.981011 (+ 7332us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:34.981021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:34.981084 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:34.981221 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:34.981222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:34.981223 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:34.981279 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.981280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.982116 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.982116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.983121 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:34.983123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:34.983565 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:34.983565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:34.983612 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:34.983623 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:34.983785 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:34.983786 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:34.983819 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 20:55:34.983849 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7299,"thread_start_us":70,"threads_started":1}
I20260619 20:55:34.986079 8069 raft_consensus.cc:1261] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Refusing update from remote peer 83e9a2a3fd6a4c8b8ef1149f57482b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.986240 8349 raft_consensus.cc:1261] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Refusing update from remote peer 83e9a2a3fd6a4c8b8ef1149f57482b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:34.986346 8470 consensus_queue.cc:1035] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.986440 8426 consensus_queue.cc:1035] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:34.988011 8433 mvcc.cc:205] Tried to move back new op lower bound from 7298672783301656576 to 7298672782311079936. Current Snapshot: MvccSnapshot[applied={T|T < 7298672783301656576}]
I20260619 20:55:34.998036 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:34.998112 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:34.998656 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Sending DeleteTablet for 3 replicas of tablet 208e7034e6e14966b3ccab615337f48d
I20260619 20:55:34.998879 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 208e7034e6e14966b3ccab615337f48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 20:55:34.998930 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 208e7034e6e14966b3ccab615337f48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260619 20:55:34.999073 8445 tablet_replica.cc:330] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: stopping tablet replica
I20260619 20:55:34.999130 8445 raft_consensus.cc:2229] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:34.999168 8445 raft_consensus.cc:2258] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:34.999228 8443 tablet_replica.cc:330] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: stopping tablet replica
I20260619 20:55:34.999286 8443 raft_consensus.cc:2229] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:34.999318 8443 raft_consensus.cc:2258] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.000973 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 208e7034e6e14966b3ccab615337f48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60934
I20260619 20:55:35.001078 8444 tablet_replica.cc:330] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: stopping tablet replica
I20260619 20:55:35.001147 8444 raft_consensus.cc:2229] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:35.001230 8444 raft_consensus.cc:2258] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.001318 8443 ts_tablet_manager.cc:1905] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.001595 8444 ts_tablet_manager.cc:1905] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.001698 8445 ts_tablet_manager.cc:1905] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.002904 8443 ts_tablet_manager.cc:1918] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:35.002957 8443 log.cc:1196] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/208e7034e6e14966b3ccab615337f48d
I20260619 20:55:35.003119 8444 ts_tablet_manager.cc:1918] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:35.003140 8445 ts_tablet_manager.cc:1918] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:35.003170 8444 log.cc:1196] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/208e7034e6e14966b3ccab615337f48d
I20260619 20:55:35.003175 8443 ts_tablet_manager.cc:1939] T 208e7034e6e14966b3ccab615337f48d P d5585e0123f44618883d45fa602b1b3c: Deleting consensus metadata
I20260619 20:55:35.003185 8445 log.cc:1196] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/208e7034e6e14966b3ccab615337f48d
I20260619 20:55:35.003381 8444 ts_tablet_manager.cc:1939] T 208e7034e6e14966b3ccab615337f48d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting consensus metadata
I20260619 20:55:35.003388 8445 ts_tablet_manager.cc:1939] T 208e7034e6e14966b3ccab615337f48d P e9067a5330ad4bfb814116f1c0e12793: Deleting consensus metadata
I20260619 20:55:35.003505 7896 catalog_manager.cc:4841] TS d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907): tablet 208e7034e6e14966b3ccab615337f48d (table test-table [id=85aa78e90a4746eaa2023a2e54a1244e]) successfully deleted
I20260619 20:55:35.003664 7898 catalog_manager.cc:4841] TS e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789): tablet 208e7034e6e14966b3ccab615337f48d (table test-table [id=85aa78e90a4746eaa2023a2e54a1244e]) successfully deleted
I20260619 20:55:35.003825 7899 catalog_manager.cc:4841] TS 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127): tablet 208e7034e6e14966b3ccab615337f48d (table test-table [id=85aa78e90a4746eaa2023a2e54a1244e]) successfully deleted
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[7867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:55:35.021898 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:35.016049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.98.62:45977)
0619 20:55:35.016114 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.016116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.016225 (+ 109us) server_negotiation.cc:407] Connection header received
0619 20:55:35.016295 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.016296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.016304 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.016351 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:35.017019 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.017501 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.018589 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.018770 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.018822 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.021814 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43890: BlockingRecv error: recv got EOF from 127.0.0.1:43890 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[7867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:55:35.029939 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.022863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.98.62:45977)
0619 20:55:35.022900 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.022902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.022954 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:55:35.023038 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.023039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.023047 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.023116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:35.023663 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.024150 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.025109 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.025278 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.025328 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.027528 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:35.027534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:35.027534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:35.027540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:35.028878 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:35.029207 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:35.029208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:35.029209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:35.029265 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:35.029461 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:35.029462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:35.029463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:35.029569 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:35.029663 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.029764 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.029839 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:35.031636 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:35.031844 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:35.033145 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 74f17dce98994879b8d8ce64371342c0 (DEFAULT_TABLE table=test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.033197 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 74f17dce98994879b8d8ce64371342c0 (DEFAULT_TABLE table=test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.033296 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f17dce98994879b8d8ce64371342c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.033337 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f17dce98994879b8d8ce64371342c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.034349 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 74f17dce98994879b8d8ce64371342c0 (DEFAULT_TABLE table=test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.034467 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f17dce98994879b8d8ce64371342c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.035343 8421 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Bootstrap starting.
I20260619 20:55:35.035977 8421 tablet_bootstrap.cc:654] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.036512 8420 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Bootstrap starting.
I20260619 20:55:35.036856 8421 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: No bootstrap required, opened a new log
I20260619 20:55:35.036899 8421 ts_tablet_manager.cc:1397] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:35.036995 8420 tablet_bootstrap.cc:654] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.037071 8421 raft_consensus.cc:348] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.037166 8421 raft_consensus.cc:374] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.037192 8421 raft_consensus.cc:729] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.037246 8421 consensus_queue.cc:260] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.037364 8421 ts_tablet_manager.cc:1428] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:35.037369 8423 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Bootstrap starting.
I20260619 20:55:35.037642 8423 tablet_bootstrap.cc:654] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.038146 8423 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: No bootstrap required, opened a new log
I20260619 20:55:35.038197 8423 ts_tablet_manager.cc:1397] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:35.038354 8423 raft_consensus.cc:348] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.038406 8423 raft_consensus.cc:374] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.038419 8423 raft_consensus.cc:729] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.038465 8423 consensus_queue.cc:260] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.038553 8423 ts_tablet_manager.cc:1428] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:35.039273 8420 tablet_bootstrap.cc:492] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: No bootstrap required, opened a new log
I20260619 20:55:35.039343 8420 ts_tablet_manager.cc:1397] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:55:35.039500 8420 raft_consensus.cc:348] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.039574 8420 raft_consensus.cc:374] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.039595 8420 raft_consensus.cc:729] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.039635 8420 consensus_queue.cc:260] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.039736 8420 ts_tablet_manager.cc:1428] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:35.189733 8427 raft_consensus.cc:482] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:35.189841 8427 raft_consensus.cc:504] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.190176 8427 leader_election.cc:288] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127)
I20260619 20:55:35.194052 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.190413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35515 (local address 127.0.98.1:37907)
0619 20:55:35.190459 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.190463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.190595 (+ 132us) server_negotiation.cc:407] Connection header received
0619 20:55:35.190698 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.190699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.190710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.190739 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:35.190975 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.191695 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193205 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.193631 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193708 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.193721 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.193927 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.193973 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:55:35.194059 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.190360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37907 (local address 127.0.98.3:35515)
0619 20:55:35.190530 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:35.190542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:35.190616 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:35.190816 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:35.190817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:35.190819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:35.190870 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.190871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.191810 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.191811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.193085 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.193088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.193728 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.193818 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.193823 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:55:35.193905 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:55:35.193954 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":41,"thread_start_us":67,"threads_started":1}
I20260619 20:55:35.194115 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.190526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53791 (local address 127.0.98.2:37127)
0619 20:55:35.190566 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.190568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.190592 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:55:35.190700 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.190702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.190712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.190766 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:35.191188 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.191661 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193165 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.193750 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193824 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.193835 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.193993 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.194038 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:35.194361 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.190370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.98.3:53791)
0619 20:55:35.190530 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:35.190542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:35.190588 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:35.190775 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:35.190776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:35.190778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:35.190855 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.190859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.191741 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.191741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.193085 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.193088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.193841 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.193842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.193912 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.193914 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:35.194195 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 20:55:35.194278 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":32,"thread_start_us":59,"threads_started":1}
I20260619 20:55:35.194528 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f17dce98994879b8d8ce64371342c0" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c" is_pre_election: true
I20260619 20:55:35.194634 8069 raft_consensus.cc:2454] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 0.
I20260619 20:55:35.194669 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f17dce98994879b8d8ce64371342c0" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" is_pre_election: true
I20260619 20:55:35.194772 8209 raft_consensus.cc:2454] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 0.
I20260619 20:55:35.194798 8280 leader_election.cc:302] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5585e0123f44618883d45fa602b1b3c, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:35.194938 8427 raft_consensus.cc:2781] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:35.194978 8427 raft_consensus.cc:482] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:35.195003 8427 raft_consensus.cc:3037] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.195643 8427 raft_consensus.cc:504] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.195779 8427 leader_election.cc:288] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 election: Requested vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127)
I20260619 20:55:35.195920 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f17dce98994879b8d8ce64371342c0" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c"
I20260619 20:55:35.195953 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f17dce98994879b8d8ce64371342c0" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
I20260619 20:55:35.196008 8069 raft_consensus.cc:3037] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.196017 8209 raft_consensus.cc:3037] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.196590 8209 raft_consensus.cc:2454] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 1.
I20260619 20:55:35.196696 8069 raft_consensus.cc:2454] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 1.
I20260619 20:55:35.196759 8283 leader_election.cc:302] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:35.196851 8427 raft_consensus.cc:2781] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:35.196969 8427 raft_consensus.cc:686] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 LEADER]: Becoming Leader. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Running, Role: LEADER
I20260619 20:55:35.197036 8427 consensus_queue.cc:237] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.197687 7911 catalog_manager.cc:5495] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e9067a5330ad4bfb814116f1c0e12793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: HEALTHY } } }
I20260619 20:55:35.223587 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.221032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.98.3:36789)
0619 20:55:35.221077 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.221079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.221088 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:35.221164 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.221165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.221191 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.221262 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:35.221382 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.221916 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.222726 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.222977 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.223055 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.223215 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:35.223349 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:35.223430 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.223486 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.223504 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:35.224558 8209 raft_consensus.cc:1261] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Refusing update from remote peer e9067a5330ad4bfb814116f1c0e12793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:35.224565 8069 raft_consensus.cc:1261] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Refusing update from remote peer e9067a5330ad4bfb814116f1c0e12793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:35.224879 8491 consensus_queue.cc:1035] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:35.225010 8427 consensus_queue.cc:1035] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:35.228127 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:35.228219 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:35.228780 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Sending DeleteTablet for 3 replicas of tablet 74f17dce98994879b8d8ce64371342c0
I20260619 20:55:35.229035 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f17dce98994879b8d8ce64371342c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 20:55:35.229023 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f17dce98994879b8d8ce64371342c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60934
I20260619 20:55:35.229041 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f17dce98994879b8d8ce64371342c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260619 20:55:35.229110 8445 tablet_replica.cc:330] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: stopping tablet replica
I20260619 20:55:35.229135 8444 tablet_replica.cc:330] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: stopping tablet replica
I20260619 20:55:35.229161 8445 raft_consensus.cc:2229] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:35.229184 8444 raft_consensus.cc:2229] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:35.229223 8444 raft_consensus.cc:2258] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.229234 8445 raft_consensus.cc:2258] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.229296 8443 tablet_replica.cc:330] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: stopping tablet replica
I20260619 20:55:35.229348 8443 raft_consensus.cc:2229] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:35.229394 8443 raft_consensus.cc:2258] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.229678 8443 ts_tablet_manager.cc:1905] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.229846 8445 ts_tablet_manager.cc:1905] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.229810 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:35.229930 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:35.231074 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 3496da7bd63948cf9e51a138480baf7d (DEFAULT_TABLE table=test-table [id=ad9c7ee763a6430882573132a4a6fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.231213 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3496da7bd63948cf9e51a138480baf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.231309 8445 ts_tablet_manager.cc:1918] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:35.231321 8443 ts_tablet_manager.cc:1918] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:35.231362 8445 log.cc:1196] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/74f17dce98994879b8d8ce64371342c0
I20260619 20:55:35.231371 8443 log.cc:1196] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/74f17dce98994879b8d8ce64371342c0
I20260619 20:55:35.231593 8443 ts_tablet_manager.cc:1939] T 74f17dce98994879b8d8ce64371342c0 P d5585e0123f44618883d45fa602b1b3c: Deleting consensus metadata
I20260619 20:55:35.231597 8445 ts_tablet_manager.cc:1939] T 74f17dce98994879b8d8ce64371342c0 P e9067a5330ad4bfb814116f1c0e12793: Deleting consensus metadata
I20260619 20:55:35.232010 8444 ts_tablet_manager.cc:1905] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.232522 8420 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Bootstrap starting.
I20260619 20:55:35.232728 7898 catalog_manager.cc:4841] TS e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789): tablet 74f17dce98994879b8d8ce64371342c0 (table test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]) successfully deleted
I20260619 20:55:35.232839 8420 tablet_bootstrap.cc:654] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.232887 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 3496da7bd63948cf9e51a138480baf7d (DEFAULT_TABLE table=test-table [id=ad9c7ee763a6430882573132a4a6fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.232975 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3496da7bd63948cf9e51a138480baf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.233369 8444 ts_tablet_manager.cc:1918] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:35.233439 8444 log.cc:1196] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/74f17dce98994879b8d8ce64371342c0
I20260619 20:55:35.233654 8444 ts_tablet_manager.cc:1939] T 74f17dce98994879b8d8ce64371342c0 P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting consensus metadata
I20260619 20:55:35.233935 8421 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Bootstrap starting.
I20260619 20:55:35.234033 7899 catalog_manager.cc:4841] TS 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127): tablet 74f17dce98994879b8d8ce64371342c0 (table test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]) successfully deleted
I20260619 20:55:35.234197 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 3496da7bd63948cf9e51a138480baf7d (DEFAULT_TABLE table=test-table [id=ad9c7ee763a6430882573132a4a6fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:35.234309 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3496da7bd63948cf9e51a138480baf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:35.234354 8421 tablet_bootstrap.cc:654] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.234880 8420 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: No bootstrap required, opened a new log
I20260619 20:55:35.234930 8420 ts_tablet_manager.cc:1397] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:35.235034 8421 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: No bootstrap required, opened a new log
I20260619 20:55:35.235036 8420 raft_consensus.cc:348] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.235069 8421 ts_tablet_manager.cc:1397] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:35.235108 8420 raft_consensus.cc:374] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.235133 8420 raft_consensus.cc:729] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9a2a3fd6a4c8b8ef1149f57482b51, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.235181 8420 consensus_queue.cc:260] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.235215 8421 raft_consensus.cc:348] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.235260 8420 ts_tablet_manager.cc:1428] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:35.235283 8421 raft_consensus.cc:374] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.235304 8421 raft_consensus.cc:729] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.235345 8421 consensus_queue.cc:260] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.235425 8421 ts_tablet_manager.cc:1428] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:35.235452 8423 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: Bootstrap starting.
I20260619 20:55:35.235733 8423 tablet_bootstrap.cc:654] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:35.236128 7896 catalog_manager.cc:4841] TS d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907): tablet 74f17dce98994879b8d8ce64371342c0 (table test-table [id=a0f8355e4f3d4a0ab5ac9b4e269682cb]) successfully deleted
I20260619 20:55:35.236442 8423 tablet_bootstrap.cc:492] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: No bootstrap required, opened a new log
I20260619 20:55:35.236497 8423 ts_tablet_manager.cc:1397] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:35.236596 8423 raft_consensus.cc:348] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.236665 8423 raft_consensus.cc:374] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:35.236686 8423 raft_consensus.cc:729] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5585e0123f44618883d45fa602b1b3c, State: Initialized, Role: FOLLOWER
I20260619 20:55:35.236724 8423 consensus_queue.cc:260] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.236824 8423 ts_tablet_manager.cc:1428] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:35.242760 8496 raft_consensus.cc:482] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:35.242846 8496 raft_consensus.cc:504] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.242978 8496 leader_election.cc:288] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127)
I20260619 20:55:35.243108 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3496da7bd63948cf9e51a138480baf7d" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c" is_pre_election: true
I20260619 20:55:35.243103 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3496da7bd63948cf9e51a138480baf7d" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" is_pre_election: true
I20260619 20:55:35.243175 8069 raft_consensus.cc:2454] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 0.
I20260619 20:55:35.243180 8209 raft_consensus.cc:2454] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 0.
I20260619 20:55:35.243316 8283 leader_election.cc:302] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e9a2a3fd6a4c8b8ef1149f57482b51, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:35.243402 8496 raft_consensus.cc:2781] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:35.243434 8496 raft_consensus.cc:482] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:35.243454 8496 raft_consensus.cc:3037] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.244015 8496 raft_consensus.cc:504] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.244179 8496 leader_election.cc:288] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 election: Requested vote from peers d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907), 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127)
I20260619 20:55:35.244252 8209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3496da7bd63948cf9e51a138480baf7d" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51"
I20260619 20:55:35.244249 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3496da7bd63948cf9e51a138480baf7d" candidate_uuid: "e9067a5330ad4bfb814116f1c0e12793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5585e0123f44618883d45fa602b1b3c"
I20260619 20:55:35.244313 8069 raft_consensus.cc:3037] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.244314 8209 raft_consensus.cc:3037] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:35.244839 8069 raft_consensus.cc:2454] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 1.
I20260619 20:55:35.244974 8280 leader_election.cc:302] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5585e0123f44618883d45fa602b1b3c, e9067a5330ad4bfb814116f1c0e12793; no voters:
I20260619 20:55:35.245012 8209 raft_consensus.cc:2454] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9067a5330ad4bfb814116f1c0e12793 in term 1.
I20260619 20:55:35.245052 8496 raft_consensus.cc:2781] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:35.245111 8496 raft_consensus.cc:686] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 1 LEADER]: Becoming Leader. State: Replica: e9067a5330ad4bfb814116f1c0e12793, State: Running, Role: LEADER
I20260619 20:55:35.245152 8496 consensus_queue.cc:237] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } }
I20260619 20:55:35.245628 7911 catalog_manager.cc:5495] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e9067a5330ad4bfb814116f1c0e12793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9067a5330ad4bfb814116f1c0e12793" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: HEALTHY } } }
I20260619 20:55:35.256824 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.254064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38791 (local address 127.0.98.2:37127)
0619 20:55:35.254105 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.254107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.254246 (+ 139us) server_negotiation.cc:407] Connection header received
0619 20:55:35.254371 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.254371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.254380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.254437 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:35.254573 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.255027 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.256198 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.256381 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.256474 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.256639 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:35.256664 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:35.256703 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.256754 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.256771 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:35.257081 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.254009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37127 (local address 127.0.98.3:38791)
0619 20:55:35.254171 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:35.254181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:35.254312 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:35.254444 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:35.254445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:35.254446 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:35.254510 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.254512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.255090 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.255091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.256121 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.256125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.256464 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.256465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.256549 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.256565 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:35.256704 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:35.256705 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:35.256753 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:55:35.256792 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260619 20:55:35.258809 8069 raft_consensus.cc:1261] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Refusing update from remote peer e9067a5330ad4bfb814116f1c0e12793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:35.258803 8209 raft_consensus.cc:1261] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Refusing update from remote peer e9067a5330ad4bfb814116f1c0e12793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:35.258992 8496 consensus_queue.cc:1035] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5585e0123f44618883d45fa602b1b3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:35.259068 8427 consensus_queue.cc:1035] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e9a2a3fd6a4c8b8ef1149f57482b51" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:35.265518 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.263236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59765 (local address 127.0.98.3:36789)
0619 20:55:35.263273 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:35.263275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:35.263282 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:35.263310 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:35.263310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:35.263320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:35.263368 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:35.263517 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.264083 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.264794 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.264991 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.265061 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.265275 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:35.265313 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:35.265360 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:35.265438 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:55:35.265454 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:35.265532 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.263086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36789 (local address 127.0.98.2:59765)
0619 20:55:35.263129 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:35.263139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:35.263172 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:35.263410 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:35.263410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:35.263412 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:55:35.263458 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.263459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.264155 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.264156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.264709 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:35.264711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:35.265096 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:35.265102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:35.265156 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:35.265163 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:55:35.265383 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:55:35.265384 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:55:35.265422 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 20:55:35.265468 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 20:55:35.271502 7911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:35.271595 7911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:35.272116 7911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 922e3fd996194f40806c6d95407c5ba3: Sending DeleteTablet for 3 replicas of tablet 3496da7bd63948cf9e51a138480baf7d
I20260619 20:55:35.272328 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 3496da7bd63948cf9e51a138480baf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 20:55:35.272388 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 3496da7bd63948cf9e51a138480baf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260619 20:55:35.272413 8445 tablet_replica.cc:330] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: stopping tablet replica
I20260619 20:55:35.272454 8445 raft_consensus.cc:2229] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:35.272461 8443 tablet_replica.cc:330] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: stopping tablet replica
I20260619 20:55:35.272500 8443 raft_consensus.cc:2229] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:35.272511 8445 raft_consensus.cc:2258] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.272529 8443 raft_consensus.cc:2258] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.272576 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 3496da7bd63948cf9e51a138480baf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60934
I20260619 20:55:35.272672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7984
I20260619 20:55:35.272989 8443 ts_tablet_manager.cc:1905] T 3496da7bd63948cf9e51a138480baf7d P d5585e0123f44618883d45fa602b1b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.273219 8444 tablet_replica.cc:330] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: stopping tablet replica
I20260619 20:55:35.273273 8444 raft_consensus.cc:2229] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:35.273305 8444 raft_consensus.cc:2258] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:35.273558 8445 ts_tablet_manager.cc:1905] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.274924 8445 ts_tablet_manager.cc:1918] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:35.274982 8445 log.cc:1196] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/3496da7bd63948cf9e51a138480baf7d
I20260619 20:55:35.275013 8444 ts_tablet_manager.cc:1905] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:35.275199 8445 ts_tablet_manager.cc:1939] T 3496da7bd63948cf9e51a138480baf7d P e9067a5330ad4bfb814116f1c0e12793: Deleting consensus metadata
I20260619 20:55:35.275478 7898 catalog_manager.cc:4841] TS e9067a5330ad4bfb814116f1c0e12793 (127.0.98.3:36789): tablet 3496da7bd63948cf9e51a138480baf7d (table test-table [id=ad9c7ee763a6430882573132a4a6fbc1]) successfully deleted
I20260619 20:55:35.276367 8444 ts_tablet_manager.cc:1918] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:55:35.276428 8444 log.cc:1196] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/3496da7bd63948cf9e51a138480baf7d
I20260619 20:55:35.276629 8444 ts_tablet_manager.cc:1939] T 3496da7bd63948cf9e51a138480baf7d P 83e9a2a3fd6a4c8b8ef1149f57482b51: Deleting consensus metadata
I20260619 20:55:35.277144 7899 catalog_manager.cc:4841] TS 83e9a2a3fd6a4c8b8ef1149f57482b51 (127.0.98.2:37127): tablet 3496da7bd63948cf9e51a138480baf7d (table test-table [id=ad9c7ee763a6430882573132a4a6fbc1]) successfully deleted
W20260619 20:55:35.282246 7896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37907 (error 108)
I20260619 20:55:35.282754 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:35.282585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37907 (local address 127.0.0.1:35848)
0619 20:55:35.282672 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:35.282710 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":52,"threads_started":1}
I20260619 20:55:35.282804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8124
W20260619 20:55:35.284137 7896 catalog_manager.cc:4584] TS d5585e0123f44618883d45fa602b1b3c (127.0.98.1:37907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3496da7bd63948cf9e51a138480baf7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37907: connect: Connection refused (error 111)
I20260619 20:55:35.289131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8264
I20260619 20:55:35.293190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 7880
2026-06-19T20:55:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2656 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[8511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[8511](info): set up 2 sockets
Jun 19 20:55:35 dist-test-slave-8bgz krb5kdc[8511](info): commencing operation
krb5kdc: starting...
W20260619 20:55:36.330735 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:36 dist-test-slave-8bgz krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:36.445032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:36.523934 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:36.524103 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:36.524122 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:36.525164 8524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:36.525207 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:36.525219 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:36.525230 8524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:36.525290 8524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:36.526823 8524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:36.527048 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:36.527259 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:36.531949 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:36.531962 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:36.532027 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:36.532084 8524 server_base.cc:1034] running on GCE node
I20260619 20:55:36.532353 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:36.532617 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:36.533797 8524 hybrid_clock.cc:648] HybridClock initialized: now 1781902536533782 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T20:55:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1249 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:36 dist-test-slave-8bgz krb5kdc[8537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:36 dist-test-slave-8bgz krb5kdc[8537](info): set up 2 sockets
Jun 19 20:55:36 dist-test-slave-8bgz krb5kdc[8537](info): commencing operation
krb5kdc: starting...
W20260619 20:55:37.578699 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:37 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:37.688824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:37.765957 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:37.766140 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:37.766160 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:37.767324 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:37.767364 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:37.767379 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:37.767426 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:37.767457 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:37.768978 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:37.769207 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:37.769421 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:37.773958 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:37.773962 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:37.773988 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:37.774333 8550 server_base.cc:1034] running on GCE node
I20260619 20:55:37.774564 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:37.774791 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:37.775919 8550 hybrid_clock.cc:648] HybridClock initialized: now 1781902537775908 us; error 28 us; skew 500 ppm
Jun 19 20:55:37 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:37.782404 8550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:37.782882 8550 webserver.cc:460] Webserver started at http://127.0.98.62:42029/ using document root <none> and password file <none>
I20260619 20:55:37.783121 8550 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:37.783186 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:37.783315 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:37.784338 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "46afca7eee484e05baeca36a042c8445"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "2170c546c2d7cc47daf32e119e31f03f"
server_key_iv: "1860255fd3ac483c9c94a42d89d66a17"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.784694 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "46afca7eee484e05baeca36a042c8445"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "2170c546c2d7cc47daf32e119e31f03f"
server_key_iv: "1860255fd3ac483c9c94a42d89d66a17"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.785957 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:55:37.786706 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:37.786865 8550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:37.786962 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "46afca7eee484e05baeca36a042c8445"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "2170c546c2d7cc47daf32e119e31f03f"
server_key_iv: "1860255fd3ac483c9c94a42d89d66a17"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.787041 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:37.802907 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:37.804617 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:37.804773 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:37.808604 8550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36983
I20260619 20:55:37.808640 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36983 every 8 connection(s)
I20260619 20:55:37.809139 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:37.809727 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:37.811940 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Bootstrap starting.
I20260619 20:55:37.812448 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:37.812683 8618 log.cc:824] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:37.812978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8550
I20260619 20:55:37.813088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:37.813277 392 external_mini_cluster.cc:1428] Setting key 0b5aef6ce8fde66df0d9043bb41bda15
I20260619 20:55:37.813409 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: No bootstrap required, opened a new log
I20260619 20:55:37.816088 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:37.816242 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:37.816272 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46afca7eee484e05baeca36a042c8445, State: Initialized, Role: FOLLOWER
I20260619 20:55:37.816391 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:37.816476 8618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:37.816504 8618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:37.816545 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:37.817155 8618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:37.817242 8618 leader_election.cc:302] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46afca7eee484e05baeca36a042c8445; no voters:
I20260619 20:55:37.817404 8618 leader_election.cc:288] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:37.817430 8623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:37.817548 8623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 LEADER]: Becoming Leader. State: Replica: 46afca7eee484e05baeca36a042c8445, State: Running, Role: LEADER
I20260619 20:55:37.817641 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:37.817734 8623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:37.818030 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46afca7eee484e05baeca36a042c8445. Latest consensus state: current_term: 1 leader_uuid: "46afca7eee484e05baeca36a042c8445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } } }
I20260619 20:55:37.818186 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:37.818819 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46afca7eee484e05baeca36a042c8445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } } }
I20260619 20:55:37.818638 8632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:37.819290 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:37.820216 8632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:55:37 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:37.821561 8632 catalog_manager.cc:1347] Generated new cluster ID: aab7b6fa815b44449b4b159f8a2afc5e
I20260619 20:55:37.821609 8632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:37.824538 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:37.813733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.98.62:36983)
0619 20:55:37.813917 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:37.813926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:37.813958 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:55:37.814748 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:37.814756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:37.814815 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:37.814959 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:37.815504 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:37.816852 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:37.817775 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:37.818366 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:37.818480 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:37.821080 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:37.821083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:37.821084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:37.821094 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:37.822939 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:37.823336 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:37.823338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:37.823339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:37.823430 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:37.823653 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:37.823654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:37.823654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:37.823934 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:37.824069 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:37.824216 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:55:37.824344 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260619 20:55:37.862756 8632 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:37.863521 8632 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:37.867931 8632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Generated new TSK 0
I20260619 20:55:37.868108 8632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:37.892395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:37.970152 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:37.970353 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:37.970376 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:37.971553 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:37.971608 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:37.971634 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:37.973022 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:37.973194 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:37.973356 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:37.977952 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:37.977972 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:37.978065 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:37.978168 8644 server_base.cc:1034] running on GCE node
I20260619 20:55:37.978384 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:37.978554 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:37.979682 8644 hybrid_clock.cc:648] HybridClock initialized: now 1781902537979671 us; error 25 us; skew 500 ppm
Jun 19 20:55:37 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:37.981972 8644 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:37.982442 8644 webserver.cc:460] Webserver started at http://127.0.98.1:34115/ using document root <none> and password file <none>
I20260619 20:55:37.982666 8644 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:37.982728 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:37.982870 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:37.983829 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4616c8978bf4975a819238e1a10ca1c"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "4e8c758ee30b02ba3b668012d79ce1e8"
server_key_iv: "1977df976ed476299a6b7b9bf8d55a27"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.984175 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4616c8978bf4975a819238e1a10ca1c"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "4e8c758ee30b02ba3b668012d79ce1e8"
server_key_iv: "1977df976ed476299a6b7b9bf8d55a27"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.985442 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:37.986202 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:37.986382 8644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:37.986475 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b4616c8978bf4975a819238e1a10ca1c"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "4e8c758ee30b02ba3b668012d79ce1e8"
server_key_iv: "1977df976ed476299a6b7b9bf8d55a27"
server_key_version: "encryptionkey@0"
I20260619 20:55:37.986555 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.004844 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.006834 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.006987 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.007200 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.007496 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.007547 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.007586 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.007614 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.013118 8644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46481
I20260619 20:55:38.013453 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46481 every 8 connection(s)
I20260619 20:55:38.013502 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:38.016744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8644
I20260619 20:55:38.016856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:38.017056 392 external_mini_cluster.cc:1428] Setting key 64a65fa4c9212890114caa38fdb6cbc2
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:38.023761 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.014258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49085 (local address 127.0.98.62:36983)
0619 20:55:38.014303 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.014306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.014549 (+ 243us) server_negotiation.cc:407] Connection header received
0619 20:55:38.015483 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.015484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.015493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.015545 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.016095 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.016592 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.018140 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.018435 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.018497 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.020853 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:38.020854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:38.020855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:38.020862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:38.022363 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.022778 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.022779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.022779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.022835 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.023126 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.023127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.023127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.023239 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:38.023309 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:38.023575 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 20:55:38.023667 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:38.023933 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.014250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:49085)
0619 20:55:38.014401 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.014444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.015394 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.015596 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.015600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.015835 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.016025 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.016029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.016681 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.016683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.018039 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.018041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.018516 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.018517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.018670 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.019192 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.019203 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.020745 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:38.022453 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.022460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.022463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.022691 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.022898 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.022899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.022899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.023056 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.023308 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:38.023315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:38.023448 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:55:38.023551 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 20:55:38.023701 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260619 20:55:38.024385 8773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36983
I20260619 20:55:38.024495 8773 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:38.024704 8773 heartbeater.cc:511] Master 127.0.98.62:36983 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:55:38.025193 8582 ts_manager.cc:194] Registered new tserver with Master: b4616c8978bf4975a819238e1a10ca1c (127.0.98.1:46481)
I20260619 20:55:38.025790 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:38.040900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:38.120350 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.120569 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.120592 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.121644 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.121673 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.121699 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:38.123072 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.123241 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.123402 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.128046 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.128098 8778 server_base.cc:1034] running on GCE node
W20260619 20:55:38.128048 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.128048 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.128499 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.128762 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.129911 8778 hybrid_clock.cc:648] HybridClock initialized: now 1781902538129903 us; error 35 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.132200 8778 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:38.132624 8778 webserver.cc:460] Webserver started at http://127.0.98.2:42291/ using document root <none> and password file <none>
I20260619 20:55:38.132840 8778 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.132902 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.133037 8778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:38.134002 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "95d90d92bf944cc29c7d5fb5972b26a7"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "67f1611482acf6eb2a8e2b7917496352"
server_key_iv: "c4c723643d0535bb3f800e3afec7c794"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.134353 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95d90d92bf944cc29c7d5fb5972b26a7"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "67f1611482acf6eb2a8e2b7917496352"
server_key_iv: "c4c723643d0535bb3f800e3afec7c794"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.135563 8778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:38.136204 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.136364 8778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:38.136457 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "95d90d92bf944cc29c7d5fb5972b26a7"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "67f1611482acf6eb2a8e2b7917496352"
server_key_iv: "c4c723643d0535bb3f800e3afec7c794"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.136543 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.150672 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.152438 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.152592 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.152803 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.153088 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.153136 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.153170 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.153198 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.158852 8778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46055
I20260619 20:55:38.158910 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46055 every 8 connection(s)
I20260619 20:55:38.159195 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:38.165270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8778
I20260619 20:55:38.165390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:38.165607 392 external_mini_cluster.cc:1428] Setting key 4ddb4b3ea886dcc100a401533d634978
I20260619 20:55:38.168149 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.159795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44685 (local address 127.0.98.62:36983)
0619 20:55:38.159900 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.159903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.160055 (+ 152us) server_negotiation.cc:407] Connection header received
0619 20:55:38.160929 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.160930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.160939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.161004 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.161576 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.162071 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.163043 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.163222 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.163275 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.165167 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:38.165168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:38.165169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:38.165174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:38.166687 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.167156 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.167157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.167158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.167207 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.167468 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.167469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.167469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.167563 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:38.167631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:38.167943 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 20:55:38.168029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 20:55:38.168167 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.159765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:44685)
0619 20:55:38.159924 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.159962 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.160827 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.161067 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.161070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.161320 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.161494 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.161497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.162144 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.162145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.162965 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.162968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.163290 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.163291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.163388 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.163782 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.163789 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.165062 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:38.166758 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.166765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.166768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.167082 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.167267 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.167268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.167268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.167408 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.167623 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:38.167630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:38.167797 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 20:55:38.167904 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:55:38.168020 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260619 20:55:38.168599 8907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36983
I20260619 20:55:38.168707 8907 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:38.168926 8907 heartbeater.cc:511] Master 127.0.98.62:36983 requested a full tablet report, sending...
I20260619 20:55:38.169287 8582 ts_manager.cc:194] Registered new tserver with Master: 95d90d92bf944cc29c7d5fb5972b26a7 (127.0.98.2:46055)
I20260619 20:55:38.169804 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44685
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:38.189723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:38.268476 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.268687 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.268709 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.269731 8912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.269776 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.269819 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:38.271291 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.271466 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.271646 8912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.276430 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.276368 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.276459 8912 server_base.cc:1034] running on GCE node
W20260619 20:55:38.276391 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.276849 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.277088 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.278263 8912 hybrid_clock.cc:648] HybridClock initialized: now 1781902538278253 us; error 32 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.280642 8912 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:38.281054 8912 webserver.cc:460] Webserver started at http://127.0.98.3:34501/ using document root <none> and password file <none>
I20260619 20:55:38.281265 8912 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.281327 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.281463 8912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:38.282591 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "c6009a6948f34f99ad91a0156344ece2"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "69964ed2b2efd5180d8ba45099032fbc"
server_key_iv: "0594bd41ef18d663e4886a7c1de1ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.282939 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c6009a6948f34f99ad91a0156344ece2"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "69964ed2b2efd5180d8ba45099032fbc"
server_key_iv: "0594bd41ef18d663e4886a7c1de1ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.284193 8912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:38.284842 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.285000 8912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:38.285100 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "c6009a6948f34f99ad91a0156344ece2"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "69964ed2b2efd5180d8ba45099032fbc"
server_key_iv: "0594bd41ef18d663e4886a7c1de1ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.285174 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.299310 8912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.301644 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.301750 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.301918 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.302171 8912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.302239 8912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.302278 8912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.302305 8912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.308015 8912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36753
I20260619 20:55:38.308055 9040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36753 every 8 connection(s)
I20260619 20:55:38.308347 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:38.314004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 8912
I20260619 20:55:38.314121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:38.314376 392 external_mini_cluster.cc:1428] Setting key 43bc64f898c5ff3227a18e7ab3290596
I20260619 20:55:38.316757 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.308949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57707 (local address 127.0.98.62:36983)
0619 20:55:38.309007 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.309009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.309241 (+ 232us) server_negotiation.cc:407] Connection header received
0619 20:55:38.310115 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.310117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.310126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.310175 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.310765 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.311243 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.312150 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.312323 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.312370 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.314319 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:38.314321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:38.314322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:38.314330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:38.315733 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.316072 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.316072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.316073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.316119 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.316312 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.316312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.316312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.316404 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:38.316455 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:38.316649 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 20:55:38.316700 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:55:38.316991 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.308951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:57707)
0619 20:55:38.309133 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.309165 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.310034 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.310274 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.310277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.310509 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.310689 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.310693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.311314 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.311315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.312085 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.312086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.312385 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.312386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.312474 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.312843 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.312850 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.314136 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:38.315800 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.315806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.315807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.316008 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.316170 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:38.316170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:38.316171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:38.316260 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:38.316454 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:38.316458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:38.316563 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 20:55:38.316638 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:55:38.316743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260619 20:55:38.317451 9041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36983
I20260619 20:55:38.317548 9041 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:38.317735 9041 heartbeater.cc:511] Master 127.0.98.62:36983 requested a full tablet report, sending...
I20260619 20:55:38.318099 8582 ts_manager.cc:194] Registered new tserver with Master: c6009a6948f34f99ad91a0156344ece2 (127.0.98.3:36753)
I20260619 20:55:38.318513 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57707
I20260619 20:55:38.325824 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:55:38.392237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8644
I20260619 20:55:38.397219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8778
I20260619 20:55:38.401262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8912
I20260619 20:55:38.405247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 8550
I20260619 20:55:38.410076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36983
--webserver_interface=127.0.98.62
--webserver_port=42029
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:38.487841 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.488077 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.488116 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.489272 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:38.489336 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.489359 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.489389 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:38.489418 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:38.490991 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.491255 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.491468 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.496191 9056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.496269 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.496349 9050 server_base.cc:1034] running on GCE node
W20260619 20:55:38.496407 9055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.496651 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.496874 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.498032 9050 hybrid_clock.cc:648] HybridClock initialized: now 1781902538498023 us; error 32 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.500396 9050 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:38.500835 9050 webserver.cc:460] Webserver started at http://127.0.98.62:42029/ using document root <none> and password file <none>
I20260619 20:55:38.501070 9050 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.501148 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.502467 9050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:38.503077 9065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.503346 9050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:55:38.503443 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "46afca7eee484e05baeca36a042c8445"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "2170c546c2d7cc47daf32e119e31f03f"
server_key_iv: "1860255fd3ac483c9c94a42d89d66a17"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.503764 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.543674 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.545678 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.545847 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.549842 9050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36983
I20260619 20:55:38.549875 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36983 every 8 connection(s)
I20260619 20:55:38.550443 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:38.551641 9118 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:55:38.552197 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Bootstrap starting.
I20260619 20:55:38.554723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9050
I20260619 20:55:38.555063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46481
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36983
--webserver_port=34115
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:55:38.560750 9118 log.cc:824] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:38.562331 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:55:38.562606 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Bootstrap complete.
I20260619 20:55:38.564713 9118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:38.565143 9118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46afca7eee484e05baeca36a042c8445, State: Initialized, Role: FOLLOWER
I20260619 20:55:38.565332 9118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:38.565397 9118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:38.565455 9118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:38.565500 9118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:55:38.566571 9118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:38.566654 9118 leader_election.cc:302] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46afca7eee484e05baeca36a042c8445; no voters:
I20260619 20:55:38.566823 9118 leader_election.cc:288] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:55:38.566908 9122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:55:38.567047 9122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [term 2 LEADER]: Becoming Leader. State: Replica: 46afca7eee484e05baeca36a042c8445, State: Running, Role: LEADER
I20260619 20:55:38.567097 9118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:38.567140 9122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } }
I20260619 20:55:38.567361 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46afca7eee484e05baeca36a042c8445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } } }
I20260619 20:55:38.567445 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:38.567359 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46afca7eee484e05baeca36a042c8445. Latest consensus state: current_term: 2 leader_uuid: "46afca7eee484e05baeca36a042c8445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46afca7eee484e05baeca36a042c8445" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36983 } } }
I20260619 20:55:38.567613 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:38.567713 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:38.568259 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:38.568390 9127 catalog_manager.cc:1259] Loaded cluster ID: aab7b6fa815b44449b4b159f8a2afc5e
I20260619 20:55:38.568427 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:38.569764 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:38.569969 9127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46afca7eee484e05baeca36a042c8445: Loaded TSK: 0
I20260619 20:55:38.570185 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:55:38.639780 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.639979 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.640003 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.641181 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.641224 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.641252 9120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:38.642618 9120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34115
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.642853 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.643100 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.648703 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.648694 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.648680 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.649215 9120 server_base.cc:1034] running on GCE node
I20260619 20:55:38.649430 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.649679 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.650851 9120 hybrid_clock.cc:648] HybridClock initialized: now 1781902538650833 us; error 51 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.653810 9120 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:55:38.654335 9120 webserver.cc:460] Webserver started at http://127.0.98.1:34115/ using document root <none> and password file <none>
I20260619 20:55:38.654567 9120 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.654667 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.656076 9120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:55:38.656765 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.656962 9120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:38.657065 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b4616c8978bf4975a819238e1a10ca1c"
format_stamp: "Formatted at 2026-06-19 20:55:37 on dist-test-slave-8bgz"
server_key: "4e8c758ee30b02ba3b668012d79ce1e8"
server_key_iv: "1977df976ed476299a6b7b9bf8d55a27"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.657419 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.671981 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.674449 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.674615 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.674852 9120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.675155 9120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.675210 9120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.675252 9120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.675282 9120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.681862 9120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46481
I20260619 20:55:38.681921 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46481 every 8 connection(s)
I20260619 20:55:38.682300 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.691253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9120
I20260619 20:55:38.691582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46055
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36983
--webserver_port=42291
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:55:38.691771 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.683053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:42115)
0619 20:55:38.683253 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.683298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.683908 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.684566 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.684570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.685154 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.685337 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.685341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.686987 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.686988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.687756 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.687758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.688608 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.688608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.688698 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.689267 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.689274 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.691547 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260619 20:55:38.692003 9268 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36983 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36983: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260619 20:55:38.692615 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.683163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42115 (local address 127.0.98.62:36983)
0619 20:55:38.684123 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.684132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.684165 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:55:38.684288 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.684293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.684345 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.684485 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.685460 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.686885 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.687856 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.688519 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.688630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.692446 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42115: BlockingRecv error: recv got EOF from 127.0.98.1:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":93,"threads_started":1}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.698024 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.692381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:51641)
0619 20:55:38.692543 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.692555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.692600 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.692939 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.692941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.693082 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.693132 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.693134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.694062 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.694063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.694907 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.694910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.695418 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.695420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.695493 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.695986 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.695992 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.697968 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260619 20:55:38.698164 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.692433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51641 (local address 127.0.98.62:36983)
0619 20:55:38.692767 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.692769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.692777 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:38.692798 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.692799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.692811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.692908 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.693226 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.693980 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.694995 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.695330 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.695410 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.698077 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51641: BlockingRecv error: recv got EOF from 127.0.98.1:51641 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":75,"threads_started":1}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.703119 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.698293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:55863)
0619 20:55:38.698326 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.698338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.698381 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.698729 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.698730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.698866 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.698895 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.698897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.699851 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.699851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.700343 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.700345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.700857 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.700858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.700899 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.701312 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.701316 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.703077 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 20:55:38.703234 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.698295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55863 (local address 127.0.98.62:36983)
0619 20:55:38.698595 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.698597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.698607 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:38.698628 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.698629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.698642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.698710 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.698979 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.699765 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.700445 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.700781 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.700862 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.703180 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55863: BlockingRecv error: recv got EOF from 127.0.98.1:55863 (error 108)
Metrics: {"server-negotiator.queue_time_us":276}
W20260619 20:55:38.703275 9268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:55:38.773226 9273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.773408 9273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.773430 9273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.774597 9273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.774636 9273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.774662 9273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:38.776017 9273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42291
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.776261 9273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.776472 9273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.781342 9279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.781347 9281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.781438 9273 server_base.cc:1034] running on GCE node
W20260619 20:55:38.781360 9278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.781772 9273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.781955 9273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.783098 9273 hybrid_clock.cc:648] HybridClock initialized: now 1781902538783072 us; error 49 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.785473 9273 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:55:38.785892 9273 webserver.cc:460] Webserver started at http://127.0.98.2:42291/ using document root <none> and password file <none>
I20260619 20:55:38.786080 9273 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.786151 9273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.787421 9273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:38.787971 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.788156 9273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.788239 9273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "95d90d92bf944cc29c7d5fb5972b26a7"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "67f1611482acf6eb2a8e2b7917496352"
server_key_iv: "c4c723643d0535bb3f800e3afec7c794"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.788569 9273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.799257 9273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.801196 9273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.801314 9273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.801479 9273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.801717 9273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.801745 9273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.801777 9273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.801789 9273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.807570 9273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46055
I20260619 20:55:38.807638 9401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46055 every 8 connection(s)
I20260619 20:55:38.807936 9273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:55:38.814620 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.808557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54283 (local address 127.0.98.62:36983)
0619 20:55:38.808623 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.808625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.808837 (+ 212us) server_negotiation.cc:407] Connection header received
0619 20:55:38.809310 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.809311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.809323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.809378 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.810326 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.810800 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.811741 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.811915 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.811963 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.814575 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54283: BlockingRecv error: recv got EOF from 127.0.98.2:54283 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:55:38.814627 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.808552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:54283)
0619 20:55:38.808708 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.808747 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.809230 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.809436 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.809439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.810017 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.810197 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.810251 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.810870 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.810871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.811663 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.811665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.811980 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.811981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.812074 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.812483 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.812490 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.814490 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
W20260619 20:55:38.814877 9402 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36983 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36983: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 20:55:38.816633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9273
I20260619 20:55:38.816918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36753
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36983
--webserver_port=34501
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.819222 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.815103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:35689)
0619 20:55:38.815148 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.815158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.815220 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.815348 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.815349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.815473 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.815512 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.815514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.816134 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.816135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.816623 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.816624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.816950 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.816951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.817019 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.817407 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.817412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.819152 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:55:38.819275 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.815180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35689 (local address 127.0.98.62:36983)
0619 20:55:38.815236 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.815238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.815246 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:38.815268 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.815269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.815279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.815345 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.815585 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.816064 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.816713 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.816884 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.816952 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.819210 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35689: BlockingRecv error: recv got EOF from 127.0.98.2:35689 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.823444 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.819481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:58701)
0619 20:55:38.819526 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.819536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.819572 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.819696 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.819696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.819817 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.819838 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.819839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.820412 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.820413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.820881 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.820883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.821230 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.821231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.821280 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.821701 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.821705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.823371 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 20:55:38.823499 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.819517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58701 (local address 127.0.98.62:36983)
0619 20:55:38.819547 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.819548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.819559 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:38.819628 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.819629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.819638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.819692 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.819913 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.820338 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.820970 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.821151 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.821229 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.823436 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58701: BlockingRecv error: recv got EOF from 127.0.98.2:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:55:38.823678 9402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:55:38.910341 9405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:38.910530 9405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:38.910559 9405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:38.911789 9405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:38.911829 9405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:38.911880 9405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:38.913337 9405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34501
--tserver_master_addrs=127.0.98.62:36983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:38.913580 9405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:38.913793 9405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:38.918599 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.918589 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:38.918588 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:38.918772 9405 server_base.cc:1034] running on GCE node
I20260619 20:55:38.918929 9405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:38.919111 9405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:38.920256 9405 hybrid_clock.cc:648] HybridClock initialized: now 1781902538920236 us; error 36 us; skew 500 ppm
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:38.922647 9405 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:55:38.923095 9405 webserver.cc:460] Webserver started at http://127.0.98.3:34501/ using document root <none> and password file <none>
I20260619 20:55:38.923319 9405 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:38.923395 9405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:38.924640 9405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:55:38.925165 9420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.925323 9405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.925419 9405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "c6009a6948f34f99ad91a0156344ece2"
format_stamp: "Formatted at 2026-06-19 20:55:38 on dist-test-slave-8bgz"
server_key: "69964ed2b2efd5180d8ba45099032fbc"
server_key_iv: "0594bd41ef18d663e4886a7c1de1ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:38.925732 9405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:38.939970 9405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:38.941670 9405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:38.941823 9405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:38.942019 9405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:38.942373 9405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:38.942425 9405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.942463 9405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:38.942490 9405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:38.948143 9405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36753
I20260619 20:55:38.948179 9533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36753 every 8 connection(s)
I20260619 20:55:38.948503 9405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:38.951900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9405
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:55:38.956135 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.949167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34637 (local address 127.0.98.62:36983)
0619 20:55:38.949233 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.949235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.949458 (+ 223us) server_negotiation.cc:407] Connection header received
0619 20:55:38.949938 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.949940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.949952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.950005 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.950943 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.951453 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.952786 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.952999 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.953053 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.956076 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34637: BlockingRecv error: recv got EOF from 127.0.98.3:34637 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:55:38.956199 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.949168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:34637)
0619 20:55:38.949323 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.949354 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.949837 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.950099 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.950103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.950675 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.950846 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.950850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.951551 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.951552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.952708 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.952710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.953089 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.953090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.953232 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.953746 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.953753 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.956008 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260619 20:55:38.956455 9534 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36983 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36983: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 20:55:38.959815 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.952413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.98.62:36983)
0619 20:55:38.952454 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.952456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.952508 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:55:38.952598 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.952599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.952608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.952657 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.953396 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.953971 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.954915 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.955120 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.955202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.956417 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:38.956422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:38.956422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:38.956433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:38.958179 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.958595 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.958598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.958599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.958697 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:38.958898 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:38.958899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:38.958899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:38.959119 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:38.959273 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:38.959620 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 20:55:38.959729 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.961313 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.956695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:36611)
0619 20:55:38.956738 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.956747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.956785 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.957154 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.957155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.957271 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.957307 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.957308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.958159 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.958159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.958696 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.958698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.959051 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.959052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.959118 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.959439 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.959443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.961232 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:55:38.961356 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.957001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36611 (local address 127.0.98.62:36983)
0619 20:55:38.957056 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.957058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.957064 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:55:38.957080 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.957080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.957086 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.957151 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.957380 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.958081 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.958789 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.958980 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.959051 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.961302 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36611: BlockingRecv error: recv got EOF from 127.0.98.3:36611 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:38 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:38.965092 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:38.961579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:41029)
0619 20:55:38.961639 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:38.961649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:38.961686 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:38.961793 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:38.961794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:38.961914 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:38.961933 (+ 19us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.961934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.962548 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.962548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.963000 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:38.963001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.963319 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.963320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:38.963367 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.963665 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:38.963669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:38.965016 (+ 1347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 20:55:38.965126 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:38.961682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41029 (local address 127.0.98.62:36983)
0619 20:55:38.961714 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:38.961716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:38.961723 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:38.961740 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:38.961741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:38.961747 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:38.961793 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:38.962000 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.962469 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.963082 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:38.963252 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:38.963321 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:38.965075 (+ 1754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41029: BlockingRecv error: recv got EOF from 127.0.98.3:41029 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:55:38.965230 9534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:39.708250 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:39.703642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:33923)
0619 20:55:39.703787 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:39.703797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:39.703848 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:39.704014 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:39.704015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:39.704166 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:39.704210 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.704212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.704830 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.704830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.705359 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.705360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.705724 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.705725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.705762 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.706199 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:39.706259 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:39.708186 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260619 20:55:39.708381 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:39.703732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33923 (local address 127.0.98.62:36983)
0619 20:55:39.703832 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:39.703834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:39.703842 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:39.703930 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:39.703931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:39.703942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:39.704001 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:39.704272 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.704770 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.705462 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.705663 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.705717 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.708273 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33923: BlockingRecv error: recv got EOF from 127.0.98.1:33923 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:39.828689 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:39.824144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:33675)
0619 20:55:39.824269 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:39.824280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:39.824349 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:39.824473 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:39.824474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:39.824624 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:39.824667 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.824668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.825309 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.825310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.825827 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.825829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.826180 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.826181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.826271 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.826687 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:39.826692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:39.828627 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260619 20:55:39.828812 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:39.824221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33675 (local address 127.0.98.62:36983)
0619 20:55:39.824344 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:39.824346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:39.824358 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:55:39.824388 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:39.824388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:39.824398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:39.824458 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:39.824732 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.825245 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.825918 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.826113 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.826174 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.828713 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33675: BlockingRecv error: recv got EOF from 127.0.98.2:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:39 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:39.970477 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:39.965653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:49673)
0619 20:55:39.965815 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:39.965825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:39.965880 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:39.966011 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:39.966012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:39.966169 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:39.966252 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.966254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.966871 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.966872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.967493 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:39.967495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.967803 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.967804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:39.967840 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.968438 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:39.968442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:39.970409 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
W20260619 20:55:39.970582 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:39.965722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49673 (local address 127.0.98.62:36983)
0619 20:55:39.965790 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:39.965792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:39.965882 (+ 90us) server_negotiation.cc:407] Connection header received
0619 20:55:39.965938 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:39.965939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:39.965951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:39.965992 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:39.966309 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.966806 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.967559 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:39.967740 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:39.967793 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:39.970493 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49673: BlockingRecv error: recv got EOF from 127.0.98.3:49673 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:40.713001 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:40.708713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:49287)
0619 20:55:40.708849 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:40.708860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:40.708929 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:40.709057 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:40.709057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:40.709219 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:40.709267 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.709268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.709874 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.709874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.710409 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.710411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.710730 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.710730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.710768 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.711187 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:40.711191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:40.712940 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260619 20:55:40.713092 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:40.708805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49287 (local address 127.0.98.62:36983)
0619 20:55:40.708924 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:40.708926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:40.708935 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:40.708975 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:40.708975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:40.708985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:40.709042 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:40.709327 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.709812 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.710468 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.710672 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.710720 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.713023 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49287: BlockingRecv error: recv got EOF from 127.0.98.1:49287 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:40.833846 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:40.829134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:49725)
0619 20:55:40.829285 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:40.829295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:40.829403 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:40.829558 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:40.829559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:40.829765 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:40.829821 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.829823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.830621 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.830622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.831139 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.831141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.831535 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.831535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.831572 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.832016 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:40.832020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:40.833746 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260619 20:55:40.833882 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:40.829203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49725 (local address 127.0.98.62:36983)
0619 20:55:40.829260 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:40.829263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:40.829387 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:55:40.829458 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:40.829459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:40.829470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:40.829540 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:40.829900 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.830553 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.831229 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.831427 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.831522 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.833818 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49725: BlockingRecv error: recv got EOF from 127.0.98.2:49725 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:40 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:40.975028 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:40.970879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:47513)
0619 20:55:40.971023 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:40.971033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:40.971082 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:40.971208 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:40.971209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:40.971360 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:40.971407 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.971408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.972044 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.972045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.972550 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:40.972552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.972856 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.972857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:40.972896 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.973331 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:40.973335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:40.974967 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
W20260619 20:55:40.975137 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:40.970996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47513 (local address 127.0.98.62:36983)
0619 20:55:40.971047 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:40.971049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:40.971071 (+ 22us) server_negotiation.cc:407] Connection header received
0619 20:55:40.971133 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:40.971134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:40.971146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:40.971191 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:40.971466 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.971984 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.972616 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:40.972796 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:40.972844 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:40.975053 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47513: BlockingRecv error: recv got EOF from 127.0.98.3:47513 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:41.718024 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:41.713627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:40461)
0619 20:55:41.713797 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:41.713808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:41.713880 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:41.714020 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:41.714021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:41.714188 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:41.714259 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.714260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.714882 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.714883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.715450 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.715451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.715789 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.715790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.715847 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.716299 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:41.716304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:41.717968 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
W20260619 20:55:41.718132 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:41.713718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40461 (local address 127.0.98.62:36983)
0619 20:55:41.713822 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:41.713824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:41.713863 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:55:41.713933 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:41.713933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:41.713944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:41.714002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:41.714316 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.714809 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.715517 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.715726 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.715773 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.718049 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40461: BlockingRecv error: recv got EOF from 127.0.98.1:40461 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:41.839004 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:41.834561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:37017)
0619 20:55:41.834700 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:41.834711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:41.834777 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:41.834910 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:41.834911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:41.835063 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:41.835109 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.835111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.835706 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.835707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.836207 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.836208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.836527 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.836528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.836566 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.837009 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:41.837014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:41.838948 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260619 20:55:41.839145 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:41.834618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37017 (local address 127.0.98.62:36983)
0619 20:55:41.834707 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:41.834709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:41.834782 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:55:41.834825 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:41.834826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:41.834837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:41.834891 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:41.835168 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.835644 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.836279 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.836462 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.836516 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.839063 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37017: BlockingRecv error: recv got EOF from 127.0.98.2:37017 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:41 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:41.980296 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:41.975547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:37289)
0619 20:55:41.975716 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:41.975734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:41.975817 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:41.975989 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:41.975990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:41.976167 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:41.976214 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.976216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.976874 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.976875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.977625 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:41.977628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.977977 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.977978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:41.978034 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.978543 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:41.978548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:41.980241 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
W20260619 20:55:41.980406 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:41.975606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37289 (local address 127.0.98.62:36983)
0619 20:55:41.975677 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:41.975680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:41.975798 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:55:41.975891 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:41.975892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:41.975905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:41.975983 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:41.976293 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.976780 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.977702 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:41.977889 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:41.977977 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:41.980328 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37289: BlockingRecv error: recv got EOF from 127.0.98.3:37289 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:42.723066 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:42.718551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:48457)
0619 20:55:42.718702 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:42.718712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:42.718784 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:42.719042 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:42.719043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:42.719219 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:42.719273 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.719275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.719904 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.719905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.720418 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.720419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.720719 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.720719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.720762 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.721256 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:42.721262 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:42.723010 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
W20260619 20:55:42.723168 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:42.718596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48457 (local address 127.0.98.62:36983)
0619 20:55:42.718799 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:42.718801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:42.718810 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:42.718939 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:42.718940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:42.718953 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:42.719020 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:42.719351 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.719839 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.720482 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.720660 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.720707 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.723095 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48457: BlockingRecv error: recv got EOF from 127.0.98.1:48457 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:42.848995 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:42.839459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:42991)
0619 20:55:42.839627 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:42.839641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:42.839708 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:42.839842 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:42.839843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:42.839997 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:42.840043 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.840045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.840654 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.840655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.841168 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.841169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.841475 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.841476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.841512 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.847026 (+ 5514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:42.847031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:42.848938 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
W20260619 20:55:42.849074 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:42.839661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42991 (local address 127.0.98.62:36983)
0619 20:55:42.839721 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:42.839723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:42.839738 (+ 15us) server_negotiation.cc:407] Connection header received
0619 20:55:42.839763 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:42.839763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:42.839773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:42.839819 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:42.840106 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.840592 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.841242 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.841417 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.841464 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.849028 (+ 7564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42991: BlockingRecv error: recv got EOF from 127.0.98.2:42991 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:42 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:42.984889 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:42.980777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:38403)
0619 20:55:42.980917 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:42.980934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:42.980987 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:42.981122 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:42.981123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:42.981281 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:42.981328 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.981329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.981939 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.981939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.982476 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:42.982478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.982778 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.982778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:42.982814 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.983231 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:42.983235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:42.984834 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
W20260619 20:55:42.985002 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:42.980851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38403 (local address 127.0.98.62:36983)
0619 20:55:42.980902 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:42.980904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:42.980989 (+ 85us) server_negotiation.cc:407] Connection header received
0619 20:55:42.981044 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:42.981045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:42.981056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:42.981101 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:42.981385 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.981875 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.982539 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:42.982719 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:42.982768 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:42.984925 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38403: BlockingRecv error: recv got EOF from 127.0.98.3:38403 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:43.727939 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:43.723551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:35415)
0619 20:55:43.723697 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:43.723710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:43.723779 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:43.723962 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:43.723963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:43.724115 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:43.724162 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.724164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.724783 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.724783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.725307 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.725309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.725624 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.725624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.725662 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.726064 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:43.726068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:43.727830 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
W20260619 20:55:43.727980 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:43.723732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35415 (local address 127.0.98.62:36983)
0619 20:55:43.723847 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:43.723849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:43.723857 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:43.723877 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:43.723877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:43.723887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:43.723944 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:43.724225 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.724722 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.725378 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.725564 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.725614 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.727907 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35415: BlockingRecv error: recv got EOF from 127.0.98.1:35415 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:43.854158 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:43.849623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:45397)
0619 20:55:43.849819 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:43.849833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:43.849891 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:43.850040 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:43.850041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:43.850190 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:43.850309 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.850311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.851070 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.851070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.851633 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.851635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.851935 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.851935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.851973 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.852425 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:43.852430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:43.854073 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
W20260619 20:55:43.854317 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:43.849805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45397 (local address 127.0.98.62:36983)
0619 20:55:43.849873 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:43.849875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:43.849884 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:43.849957 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:43.849958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:43.849968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:43.850010 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:43.850478 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.851003 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.851704 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.851877 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.851924 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.854187 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45397: BlockingRecv error: recv got EOF from 127.0.98.2:45397 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:43 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:43.990470 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:43.985386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:43165)
0619 20:55:43.985589 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:43.985606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:43.985669 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:43.985816 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:43.985817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:43.985985 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:43.986036 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.986038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.986939 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.986940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.987548 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:43.987550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.987948 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.987949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:43.988000 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.988429 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:43.988433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:43.990340 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":119,"threads_started":1}
W20260619 20:55:43.990548 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:43.985480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43165 (local address 127.0.98.62:36983)
0619 20:55:43.985536 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:43.985538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:43.985664 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:55:43.985725 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:43.985726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:43.985738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:43.985787 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:43.986108 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.986866 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.987641 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:43.987834 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:43.987927 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:43.990484 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43165: BlockingRecv error: recv got EOF from 127.0.98.3:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:44.732946 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:44.728392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:35347)
0619 20:55:44.728530 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:44.728541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:44.728619 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:44.728783 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:44.728783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:44.728936 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:44.728984 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.728986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.729754 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.729755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.730304 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.730305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.730643 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.730643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.730682 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.731200 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:44.731205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:44.732891 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20260619 20:55:44.733059 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:44.728491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35347 (local address 127.0.98.62:36983)
0619 20:55:44.728656 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:44.728658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:44.728666 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:44.728687 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:44.728688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:44.728699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:44.728765 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:44.729074 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.729687 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.730401 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.730581 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.730634 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.732982 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35347: BlockingRecv error: recv got EOF from 127.0.98.1:35347 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:44.859061 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:44.854583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:60697)
0619 20:55:44.854701 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:44.854714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:44.854781 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:44.854949 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:44.854950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:44.855125 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:44.855171 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.855173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.855843 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.855845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.856397 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.856398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.856768 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.856769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.856839 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.857262 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:44.857266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:44.859004 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260619 20:55:44.859185 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:44.854684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60697 (local address 127.0.98.62:36983)
0619 20:55:44.854727 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:44.854730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:44.854762 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:55:44.854854 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:44.854855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:44.854867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:44.854942 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:44.855260 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.855761 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.856498 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.856687 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.856758 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.859092 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60697: BlockingRecv error: recv got EOF from 127.0.98.2:60697 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:44 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:44.995551 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:44.990929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:52757)
0619 20:55:44.991091 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:44.991102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:44.991166 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:44.991325 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:44.991326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:44.991500 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:44.991546 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.991547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.992258 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.992259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.992792 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:44.992794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.993185 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.993186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:44.993244 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.993731 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:44.993736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:44.995456 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
W20260619 20:55:44.995590 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:44.991013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52757 (local address 127.0.98.62:36983)
0619 20:55:44.991077 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:44.991079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:44.991147 (+ 68us) server_negotiation.cc:407] Connection header received
0619 20:55:44.991227 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:44.991227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:44.991241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:44.991318 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:44.991621 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.992178 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.992912 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:44.993102 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:44.993176 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:44.995528 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52757: BlockingRecv error: recv got EOF from 127.0.98.3:52757 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:45.737957 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:45.733407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:46959)
0619 20:55:45.733546 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:45.733556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:45.733607 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:45.733772 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:45.733772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:45.733925 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:45.733972 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.733973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.734611 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.734612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.735127 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.735129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.735607 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.735608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.735682 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:45.736179 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:45.736185 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:45.737862 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260619 20:55:45.738005 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:45.733485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46959 (local address 127.0.98.62:36983)
0619 20:55:45.733581 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:45.733583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:45.733595 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:55:45.733692 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:45.733692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:45.733702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:45.733759 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:45.734030 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.734550 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.735260 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.735534 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.735599 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:45.737937 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46959: BlockingRecv error: recv got EOF from 127.0.98.1:46959 (error 108)
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:45.864015 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:45.859562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:52763)
0619 20:55:45.859732 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:45.859750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:45.859806 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:45.859938 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:45.859939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:45.860118 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:45.860175 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.860177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.860884 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.860885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.861400 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.861401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.861706 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.861707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.861745 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:45.862172 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:45.862177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:45.863896 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
W20260619 20:55:45.864051 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:45.859652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52763 (local address 127.0.98.62:36983)
0619 20:55:45.859737 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:45.859739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:45.859809 (+ 70us) server_negotiation.cc:407] Connection header received
0619 20:55:45.859866 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:45.859867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:45.859878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:45.859918 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:45.860239 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.860816 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.861466 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.861643 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.861692 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:45.863973 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52763: BlockingRecv error: recv got EOF from 127.0.98.2:52763 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:45 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:46.000522 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:45.995995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:58837)
0619 20:55:45.996167 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:45.996184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:45.996278 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:45.996459 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:45.996460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:45.996634 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:45.996679 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.996681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.997335 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.997336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.997865 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:45.997867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.998247 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.998248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:45.998302 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:45.998755 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:45.998759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:46.000427 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
W20260619 20:55:46.000571 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:45.996073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58837 (local address 127.0.98.62:36983)
0619 20:55:45.996139 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:45.996142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:45.996260 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:55:45.996363 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:45.996364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:45.996381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:45.996451 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:45.996756 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.997245 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.997949 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:45.998142 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:45.998262 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:46.000506 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58837: BlockingRecv error: recv got EOF from 127.0.98.3:58837 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 20:55:46 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:46 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:46.742786 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:46.738427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:35261)
0619 20:55:46.738559 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:46.738569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:46.738614 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:46.738773 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:46.738773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:46.738926 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:46.738986 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:46.738988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.739600 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.739601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:46.740100 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:46.740102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.740401 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.740402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:46.740440 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:46.740873 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:46.740877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:46.742692 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260619 20:55:46.742846 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:46.738539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35261 (local address 127.0.98.62:36983)
0619 20:55:46.738665 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:46.738667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:46.738674 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:46.738692 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:46.738693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:46.738702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:46.738758 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:46.739048 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.739539 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.740167 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.740343 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.740389 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:46.742771 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35261: BlockingRecv error: recv got EOF from 127.0.98.1:35261 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
Jun 19 20:55:46 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:46 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:46.869315 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:46.864533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:41613)
0619 20:55:46.864700 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:46.864708 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:46.864762 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:46.864880 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:46.864880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:46.865055 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:46.865100 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:46.865102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.865968 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.865969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:46.866545 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:46.866548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.866857 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.866857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:46.866898 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:46.867385 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:46.867389 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:46.869224 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
W20260619 20:55:46.869354 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:46.864600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41613 (local address 127.0.98.62:36983)
0619 20:55:46.864676 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:46.864678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:46.864760 (+ 82us) server_negotiation.cc:407] Connection header received
0619 20:55:46.864812 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:46.864813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:46.864823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:46.864860 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:46.865156 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.865887 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.866615 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:46.866797 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:46.866850 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:46.869294 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41613: BlockingRecv error: recv got EOF from 127.0.98.2:41613 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:47.005825 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:47.001039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:54431)
0619 20:55:47.001167 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:47.001180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:47.001249 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:47.001398 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:47.001399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:47.001565 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:47.001612 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.001614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.002304 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.002305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.002865 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.002867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.003187 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.003188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.003232 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.003796 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:47.003800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:47.005729 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260619 20:55:47.005901 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:47.001145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54431 (local address 127.0.98.62:36983)
0619 20:55:47.001206 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:47.001208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:47.001235 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:55:47.001303 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:47.001304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:47.001323 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:47.001386 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:47.001694 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.002183 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.002948 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.003123 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.003173 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.005814 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54431: BlockingRecv error: recv got EOF from 127.0.98.3:54431 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:47.747507 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:47.743281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:48211)
0619 20:55:47.743429 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:47.743440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:47.743502 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:47.743623 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:47.743624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:47.743769 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:47.743814 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.743815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.744404 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.744404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.744890 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.744891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.745181 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.745182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.745218 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.745636 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:47.745640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:47.747453 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
W20260619 20:55:47.747615 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:47.743339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48211 (local address 127.0.98.62:36983)
0619 20:55:47.743467 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:47.743469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:47.743487 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:55:47.743551 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:47.743551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:47.743560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:47.743606 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:47.743870 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.744333 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.744953 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.745123 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.745170 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.747540 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48211: BlockingRecv error: recv got EOF from 127.0.98.1:48211 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:47 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:47.878065 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:47.869750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:44555)
0619 20:55:47.869911 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:47.869920 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:47.869965 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:47.870102 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:47.870103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:47.870303 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:47.870379 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.870380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.871000 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.871000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.871518 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:47.871520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.871825 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.871826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:47.871865 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.876287 (+ 4422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:47.876293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:47.878010 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
W20260619 20:55:47.878248 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:47.869916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44555 (local address 127.0.98.62:36983)
0619 20:55:47.869992 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:47.869994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:47.870004 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:47.870036 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:47.870037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:47.870047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:47.870098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:47.870445 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.870935 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.871581 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:47.871764 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:47.871812 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:47.878129 (+ 6317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44555: BlockingRecv error: recv got EOF from 127.0.98.2:44555 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:48.011226 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:48.006404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:48531)
0619 20:55:48.006548 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:48.006563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:48.006616 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:48.006780 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:48.006781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:48.006939 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:48.006987 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.006989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.007865 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.007866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.008452 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.008454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.008782 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.008783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.008820 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.009319 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:48.009324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:48.011167 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
W20260619 20:55:48.011329 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:48.006473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48531 (local address 127.0.98.62:36983)
0619 20:55:48.006518 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:48.006521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:48.006620 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:55:48.006662 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:48.006662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:48.006677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:48.006737 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:48.007059 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.007783 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.008535 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.008715 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.008774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.011256 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48531: BlockingRecv error: recv got EOF from 127.0.98.3:48531 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:48.755005 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:48.747945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:51187)
0619 20:55:48.748160 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:48.748170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:48.748224 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:48.748350 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:48.748351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:48.748503 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:48.748551 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.748552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.749188 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.749188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.749720 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.749721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.750045 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.750045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.750080 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.753281 (+ 3201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:48.753286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:48.754937 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
W20260619 20:55:48.755100 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:48.748031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51187 (local address 127.0.98.62:36983)
0619 20:55:48.748161 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:48.748163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:48.748209 (+ 46us) server_negotiation.cc:407] Connection header received
0619 20:55:48.748271 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:48.748271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:48.748281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:48.748326 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:48.748621 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.749131 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.749792 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.749971 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.750036 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.755023 (+ 4987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51187: BlockingRecv error: recv got EOF from 127.0.98.1:51187 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:48 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:48.882543 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:48.878495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:44735)
0619 20:55:48.878639 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:48.878650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:48.878711 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:48.878830 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:48.878831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:48.878982 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:48.879029 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.879030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.879623 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.879624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.880118 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:48.880120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.880402 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.880403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:48.880437 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.880865 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:48.880869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:48.882489 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260619 20:55:48.882705 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:48.878536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44735 (local address 127.0.98.62:36983)
0619 20:55:48.878604 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:48.878606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:48.878721 (+ 115us) server_negotiation.cc:407] Connection header received
0619 20:55:48.878747 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:48.878747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:48.878757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:48.878804 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:48.879088 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.879563 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.880183 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:48.880344 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:48.880387 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:48.882597 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44735: BlockingRecv error: recv got EOF from 127.0.98.2:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:49.016362 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:49.011681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:45761)
0619 20:55:49.011813 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:49.011823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:49.011891 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:49.012054 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:49.012055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:49.012223 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:49.012270 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.012272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.013076 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.013077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.013625 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.013627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.013989 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.013990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.014046 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.014543 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:49.014548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:49.016281 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
W20260619 20:55:49.016498 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:49.011830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45761 (local address 127.0.98.62:36983)
0619 20:55:49.011893 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:49.011895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:49.011904 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:49.011950 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:49.011951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:49.011965 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:49.012041 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:49.012346 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.012968 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.013725 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.013918 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.013992 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.016400 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45761: BlockingRecv error: recv got EOF from 127.0.98.3:45761 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:49.759788 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:49.755492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:59473)
0619 20:55:49.755641 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:49.755651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:49.755705 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:49.755842 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:49.755843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:49.755999 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:49.756047 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.756048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.756659 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.756659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.757161 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.757162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.757471 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.757471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.757509 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.757955 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:49.757960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:49.759689 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260619 20:55:49.759917 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:49.755594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59473 (local address 127.0.98.62:36983)
0619 20:55:49.755691 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:49.755693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:49.755703 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:49.755755 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:49.755756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:49.755766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:49.755823 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:49.756108 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.756596 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.757225 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.757409 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.757457 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.759797 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59473: BlockingRecv error: recv got EOF from 127.0.98.1:59473 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:49 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:49.887683 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:49.883171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:38335)
0619 20:55:49.883351 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:49.883362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:49.883469 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:49.883619 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:49.883619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:49.883804 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:49.883847 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.883849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.884515 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.884517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.885035 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:49.885037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.885393 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.885394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:49.885452 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.885911 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:49.885915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:49.887589 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
W20260619 20:55:49.887728 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:49.883256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38335 (local address 127.0.98.62:36983)
0619 20:55:49.883342 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:49.883344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:49.883463 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:55:49.883526 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:49.883527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:49.883539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:49.883614 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:49.883938 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.884426 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.885130 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:49.885316 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:49.885390 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:49.887664 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38335: BlockingRecv error: recv got EOF from 127.0.98.2:38335 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:50.021463 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:50.016948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:35801)
0619 20:55:50.017104 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:50.017115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:50.017173 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:50.017317 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:50.017317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:50.017490 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:50.017536 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.017538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.018153 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.018153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.018690 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.018692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.019009 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.019010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.019048 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.019554 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:50.019558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:50.021405 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260619 20:55:50.021605 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:50.017058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35801 (local address 127.0.98.62:36983)
0619 20:55:50.017116 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:50.017118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:50.017179 (+ 61us) server_negotiation.cc:407] Connection header received
0619 20:55:50.017222 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:50.017222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:50.017236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:50.017294 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:50.017597 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.018085 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.018762 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.018944 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.018997 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.021522 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35801: BlockingRecv error: recv got EOF from 127.0.98.3:35801 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:50.765125 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:50.760470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:41287)
0619 20:55:50.760621 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:50.760632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:50.760690 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:50.760850 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:50.760851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:50.761002 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:50.761048 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.761049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.761728 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.761729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.762295 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.762297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.762833 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.762834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.762899 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.763372 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:50.763376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:50.765070 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260619 20:55:50.765265 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:50.760554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41287 (local address 127.0.98.62:36983)
0619 20:55:50.760678 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:50.760679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:50.760688 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:50.760761 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:50.760761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:50.760771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:50.760831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:50.761105 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.761651 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.762405 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.762737 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.762822 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.765156 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41287: BlockingRecv error: recv got EOF from 127.0.98.1:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:50 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:50.892995 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:50.888212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:35489)
0619 20:55:50.888429 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:50.888440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:50.888495 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:50.888634 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:50.888634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:50.888801 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:50.888848 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.888850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.889511 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.889512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.890035 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:50.890036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.890368 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.890369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:50.890435 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.891130 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:50.891134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:50.892904 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":154,"threads_started":1}
W20260619 20:55:50.893039 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:50.888399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35489 (local address 127.0.98.62:36983)
0619 20:55:50.888473 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:50.888475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:50.888484 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:50.888550 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:50.888551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:50.888568 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:50.888616 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:50.888952 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.889442 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.890107 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:50.890302 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:50.890351 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:50.892976 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35489: BlockingRecv error: recv got EOF from 127.0.98.2:35489 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:51.026424 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:51.021905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:57073)
0619 20:55:51.022084 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:51.022095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:51.022152 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:51.022354 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:51.022355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:51.022515 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:51.022561 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.022562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.023164 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.023165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.023676 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.023677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.024026 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.024026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.024065 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.024540 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:51.024544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:51.026328 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
W20260619 20:55:51.026466 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:51.021977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57073 (local address 127.0.98.62:36983)
0619 20:55:51.022043 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:51.022045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:51.022155 (+ 110us) server_negotiation.cc:407] Connection header received
0619 20:55:51.022267 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:51.022268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:51.022282 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:51.022333 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:51.022623 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.023100 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.023761 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.023952 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.024013 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.026405 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57073: BlockingRecv error: recv got EOF from 127.0.98.3:57073 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:51.770463 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:51.765629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:47705)
0619 20:55:51.765767 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:51.765782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:51.765829 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:51.765957 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:51.765958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:51.766125 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:51.766172 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.766174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.766926 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.766927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.767513 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.767515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.767909 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.767910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.767950 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.768356 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:51.768360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:51.770398 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260619 20:55:51.770553 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:51.765688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47705 (local address 127.0.98.62:36983)
0619 20:55:51.765821 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:51.765823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:51.765833 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:51.765869 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:51.765870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:51.765879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:51.765940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:51.766257 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.766862 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.767601 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.767849 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.767901 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.770482 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47705: BlockingRecv error: recv got EOF from 127.0.98.1:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:51 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:51.897796 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:51.893425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:35983)
0619 20:55:51.893562 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:51.893572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:51.893620 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:51.893730 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:51.893730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:51.893880 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:51.893925 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.893926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.894610 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.894611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.895132 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:51.895134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.895542 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.895543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:51.895581 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.896041 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:51.896045 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:51.897728 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260619 20:55:51.897951 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:51.893557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35983 (local address 127.0.98.62:36983)
0619 20:55:51.893610 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:51.893612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:51.893621 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:51.893677 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:51.893677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:51.893687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:51.893725 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:51.893982 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.894535 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.895241 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:51.895465 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:51.895537 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:51.897853 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35983: BlockingRecv error: recv got EOF from 127.0.98.2:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:52.031584 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:52.026893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:49311)
0619 20:55:52.027025 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:52.027043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:52.027098 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:52.027230 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:52.027231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:52.027388 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:52.027435 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.027436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.028139 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.028139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.028718 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.028720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.029045 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.029046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.029104 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.029717 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:52.029723 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:52.031510 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260619 20:55:52.031648 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:52.026973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49311 (local address 127.0.98.62:36983)
0619 20:55:52.027033 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:52.027035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:52.027096 (+ 61us) server_negotiation.cc:407] Connection header received
0619 20:55:52.027156 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:52.027157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:52.027170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:52.027212 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:52.027497 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.028039 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.028788 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.028965 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.029027 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.031601 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49311: BlockingRecv error: recv got EOF from 127.0.98.3:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:52.775305 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:52.770942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:47171)
0619 20:55:52.771075 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:52.771093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:52.771138 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:52.771278 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:52.771279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:52.771434 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:52.771481 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.771482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.772091 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.772091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.772597 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.772598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.772909 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.772909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.772948 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.773380 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:52.773384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:52.775193 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260619 20:55:52.775362 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:52.770989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47171 (local address 127.0.98.62:36983)
0619 20:55:52.771111 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:52.771117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:52.771126 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:52.771193 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:52.771194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:52.771204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:52.771262 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:52.771541 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.772028 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.772664 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.772843 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.772891 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.775287 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47171: BlockingRecv error: recv got EOF from 127.0.98.1:47171 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:52 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:52.907105 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:52.898452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:57631)
0619 20:55:52.898608 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:52.898619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:52.898694 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:52.898835 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:52.898836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:52.899050 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:52.899097 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.899099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.899705 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.899705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.900214 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:52.900215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.900519 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.900520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:52.900558 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.905287 (+ 4729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:52.905292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:52.907032 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
W20260619 20:55:52.907204 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:52.898501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57631 (local address 127.0.98.62:36983)
0619 20:55:52.898548 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:52.898552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:52.898710 (+ 158us) server_negotiation.cc:407] Connection header received
0619 20:55:52.898745 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:52.898746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:52.898759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:52.898818 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:52.899162 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.899638 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.900282 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:52.900455 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:52.900502 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:52.907118 (+ 6616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57631: BlockingRecv error: recv got EOF from 127.0.98.2:57631 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:53.036424 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:53.032059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.3:44023)
0619 20:55:53.032197 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:53.032215 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:53.032276 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:53.032399 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:53.032400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:53.032559 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:53.032607 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.032609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.033220 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.033221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.033724 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.033726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.034029 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.034029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.034067 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.034586 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:53.034591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:53.036316 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
W20260619 20:55:53.036461 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:53.032123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44023 (local address 127.0.98.62:36983)
0619 20:55:53.032201 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:53.032203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:53.032260 (+ 57us) server_negotiation.cc:407] Connection header received
0619 20:55:53.032317 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:53.032318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:53.032330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:53.032378 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:53.032668 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.033158 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.033788 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.033969 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.034017 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.036400 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44023: BlockingRecv error: recv got EOF from 127.0.98.3:44023 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:53.780404 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:53.775863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.1:56181)
0619 20:55:53.776036 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:53.776047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:53.776093 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:53.776267 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:53.776268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:53.776424 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:53.776470 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.776471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.777203 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.777204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.777706 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.777707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.778071 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.778072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.778109 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.778607 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:53.778613 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:53.780339 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260619 20:55:53.780546 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:53.775978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56181 (local address 127.0.98.62:36983)
0619 20:55:53.776122 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:53.776125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:53.776134 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:53.776157 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:53.776158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:53.776170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:53.776255 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:53.776541 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.777128 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.777787 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.778000 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.778066 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.780445 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56181: BlockingRecv error: recv got EOF from 127.0.98.1:56181 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:55:53 dist-test-slave-8bgz krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:55:53.912271 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:53.907615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36983 (local address 127.0.98.2:33867)
0619 20:55:53.907742 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:53.907753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:53.907825 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:53.908003 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:53.908003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:53.908155 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:53.908201 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.908203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.908865 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.908865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.909406 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:53.909407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.909730 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.909730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:53.909776 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.910265 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:53.910271 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:53.912165 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36983: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260619 20:55:53.912297 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:53.907812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33867 (local address 127.0.98.62:36983)
0619 20:55:53.907882 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:53.907884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:53.907891 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:55:53.907920 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:53.907921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:53.907934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:53.907984 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:53.908282 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.908793 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.909479 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:53.909660 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:53.909711 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:53.912239 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33867: BlockingRecv error: recv got EOF from 127.0.98.2:33867 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:53.954567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9120
I20260619 20:55:53.959131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9273
I20260619 20:55:53.963259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9405
I20260619 20:55:53.967391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9050
2026-06-19T20:55:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17425 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T20:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:53Z Disabled control of system clock
I20260619 20:55:53.993686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33075
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:55:54.071463 9600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.071651 9600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.071669 9600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.072872 9600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:54.072913 9600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.072926 9600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.072939 9600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:54.072990 9600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:54.074322 9600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.074563 9600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.074857 9600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.079684 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.079771 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.079742 9600 server_base.cc:1034] running on GCE node
W20260619 20:55:54.079859 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.080116 9600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.080346 9600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.081482 9600 hybrid_clock.cc:648] HybridClock initialized: now 1781902554081465 us; error 39 us; skew 500 ppm
I20260619 20:55:54.082723 9600 webserver.cc:460] Webserver started at http://127.0.98.62:36431/ using document root <none> and password file <none>
I20260619 20:55:54.082947 9600 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.082993 9600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.083158 9600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.084138 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "9da9f288a90b40f68561690d8fa17390"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "c17ce172b9e86a0aa35ded8377f735a5"
server_key_iv: "4a2206acbd7e31f84734d16ed46c77e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.084488 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "9da9f288a90b40f68561690d8fa17390"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "c17ce172b9e86a0aa35ded8377f735a5"
server_key_iv: "4a2206acbd7e31f84734d16ed46c77e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.085693 9600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:55:54.086432 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.086593 9600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.086674 9600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "9da9f288a90b40f68561690d8fa17390"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "c17ce172b9e86a0aa35ded8377f735a5"
server_key_iv: "4a2206acbd7e31f84734d16ed46c77e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.086731 9600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.111176 9600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.111441 9600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.111543 9600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.115217 9600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33075
I20260619 20:55:54.115268 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33075 every 8 connection(s)
I20260619 20:55:54.115571 9600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:54.116257 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:54.117647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9600
I20260619 20:55:54.117722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:54.117918 392 external_mini_cluster.cc:1428] Setting key eb56cb5893c240208977c7a95ddd1f8f
I20260619 20:55:54.119096 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390: Bootstrap starting.
I20260619 20:55:54.119788 9667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:54.120069 9667 log.cc:824] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:54.120795 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390: No bootstrap required, opened a new log
I20260619 20:55:54.123021 9667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260619 20:55:54.123270 9667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:54.123334 9667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9f288a90b40f68561690d8fa17390, State: Initialized, Role: FOLLOWER
I20260619 20:55:54.123449 9667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260619 20:55:54.123533 9667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:54.123594 9667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:54.123651 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:54.123805 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.118445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.98.62:33075)
0619 20:55:54.118673 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.118692 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.118715 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:55:54.119085 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.119092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.119147 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.119304 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.119551 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.120585 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.121661 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.122054 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.122139 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.122881 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.122885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.122885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.123036 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.123100 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.123114 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.123174 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.123383 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.123466 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260619 20:55:54.124300 9667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260619 20:55:54.124423 9667 leader_election.cc:302] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da9f288a90b40f68561690d8fa17390; no voters:
I20260619 20:55:54.124578 9667 leader_election.cc:288] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:54.124606 9672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:54.124747 9672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [term 1 LEADER]: Becoming Leader. State: Replica: 9da9f288a90b40f68561690d8fa17390, State: Running, Role: LEADER
I20260619 20:55:54.124842 9672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260619 20:55:54.124866 9667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:54.125172 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da9f288a90b40f68561690d8fa17390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } } }
I20260619 20:55:54.125277 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:54.125496 9678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:54.125723 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da9f288a90b40f68561690d8fa17390. Latest consensus state: current_term: 1 leader_uuid: "9da9f288a90b40f68561690d8fa17390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da9f288a90b40f68561690d8fa17390" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } } }
I20260619 20:55:54.125800 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:54.126014 9678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:54.127391 9678 catalog_manager.cc:1347] Generated new cluster ID: c91b88311b354b999b4c83e364f84846
I20260619 20:55:54.127444 9678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:54.133107 9678 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:54.133589 9678 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:54.140722 9678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9da9f288a90b40f68561690d8fa17390: Generated new TSK 0
I20260619 20:55:54.140887 9678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:54.146229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--builtin_ntp_servers=127.0.98.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 20:55:54.224923 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.225193 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.225220 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.226532 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.226574 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.226601 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:54.228425 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.228695 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.229007 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.234117 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.234081 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.234284 9691 server_base.cc:1034] running on GCE node
W20260619 20:55:54.234094 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.234591 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.234850 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.236048 9691 hybrid_clock.cc:648] HybridClock initialized: now 1781902554236025 us; error 38 us; skew 500 ppm
I20260619 20:55:54.237339 9691 webserver.cc:460] Webserver started at http://127.0.98.1:38637/ using document root <none> and password file <none>
I20260619 20:55:54.237583 9691 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.237658 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.237819 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.238953 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "23631dd66a8a498cae892c6e27b98f68"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "21168325e536f5d80c36d17c099053c6"
server_key_iv: "11006194efa2899187cb96426be949e2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.239339 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "23631dd66a8a498cae892c6e27b98f68"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "21168325e536f5d80c36d17c099053c6"
server_key_iv: "11006194efa2899187cb96426be949e2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.240721 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:54.241446 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.241632 9691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:54.241732 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "23631dd66a8a498cae892c6e27b98f68"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "21168325e536f5d80c36d17c099053c6"
server_key_iv: "11006194efa2899187cb96426be949e2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.241815 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.251718 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.251996 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.252154 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.252383 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:54.252686 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:54.252740 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.252779 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:54.252805 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.258399 9691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40393
I20260619 20:55:54.258441 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40393 every 8 connection(s)
I20260619 20:55:54.258728 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:54.260748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9691
I20260619 20:55:54.260843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:54.261063 392 external_mini_cluster.cc:1428] Setting key 0b3ca90fcf1cdff2261cfb5623ba79ec
I20260619 20:55:54.262022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--builtin_ntp_servers=127.0.98.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 20:55:54.264165 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.259343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40571 (local address 127.0.98.62:33075)
0619 20:55:54.259404 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.259406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.259622 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:55:54.260047 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.260048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.260056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.260103 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.260693 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.261474 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.262636 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.262962 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.263035 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.263889 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.263890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.263890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.263894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.263908 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.263911 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.263955 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.264050 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.264090 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:55:54.264349 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.259327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.1:40571)
0619 20:55:54.259494 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:54.259527 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:54.259975 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:54.260148 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:54.260151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:54.260158 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:54.260611 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.260615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.261560 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.261561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.262555 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.262558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.263051 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.263051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.263151 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.263729 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:54.263729 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:54.263755 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:54.263774 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:54.263954 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:54.263959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:54.263961 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:54.264050 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:55:54.264156 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260619 20:55:54.264806 9819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260619 20:55:54.264902 9819 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:54.265064 9819 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260619 20:55:54.265609 9631 ts_manager.cc:194] Registered new tserver with Master: 23631dd66a8a498cae892c6e27b98f68 (127.0.98.1:40393)
I20260619 20:55:54.266600 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40571
W20260619 20:55:54.342065 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.342288 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.342308 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.343366 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.343397 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.343423 9822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:54.344796 9822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.344974 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.345256 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.350160 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.350149 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.350409 9822 server_base.cc:1034] running on GCE node
W20260619 20:55:54.350430 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.350682 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.350920 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.352079 9822 hybrid_clock.cc:648] HybridClock initialized: now 1781902554352067 us; error 43 us; skew 500 ppm
I20260619 20:55:54.353210 9822 webserver.cc:460] Webserver started at http://127.0.98.2:40035/ using document root <none> and password file <none>
I20260619 20:55:54.353453 9822 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.353519 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.353688 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.354743 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "4f097e9ce36c42e4a35c7af123779a87"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "e091222ef7ce37dc7debee147ced80bd"
server_key_iv: "c5c7328586aaa183c49b233bd798dbba"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.355077 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4f097e9ce36c42e4a35c7af123779a87"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "e091222ef7ce37dc7debee147ced80bd"
server_key_iv: "c5c7328586aaa183c49b233bd798dbba"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.356334 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:54.357101 9836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.357300 9822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:54.357394 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "4f097e9ce36c42e4a35c7af123779a87"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "e091222ef7ce37dc7debee147ced80bd"
server_key_iv: "c5c7328586aaa183c49b233bd798dbba"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.357478 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.365801 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.366075 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.366269 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.366472 9822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:54.366816 9822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:54.366868 9822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.366907 9822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:54.366936 9822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.372555 9822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45501
I20260619 20:55:54.372601 9949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45501 every 8 connection(s)
I20260619 20:55:54.372889 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:54.376946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9822
I20260619 20:55:54.377022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:54.377264 392 external_mini_cluster.cc:1428] Setting key cabb0804dde41df657c1c43e56c7aa97
I20260619 20:55:54.377797 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.373459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40077 (local address 127.0.98.62:33075)
0619 20:55:54.373501 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.373503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.373771 (+ 268us) server_negotiation.cc:407] Connection header received
0619 20:55:54.374344 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.374345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.374356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.374407 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.374976 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.375449 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.376350 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.376520 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.376567 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.377442 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.377444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.377444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.377451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.377470 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.377474 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.377526 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.377703 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.377742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:54.378060 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.373441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.2:40077)
0619 20:55:54.373643 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:54.373681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:54.374171 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:54.374457 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:54.374460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:54.374468 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:54.374909 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.374912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.375522 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.375523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.376280 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.376282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.376585 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.376586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.376690 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.377258 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:54.377259 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:54.377294 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:54.377322 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:54.377610 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:54.377616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:54.377618 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:54.377703 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:55:54.377815 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260619 20:55:54.378324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--builtin_ntp_servers=127.0.98.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 20:55:54.378472 9950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260619 20:55:54.378582 9950 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:54.378754 9950 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260619 20:55:54.379137 9631 ts_manager.cc:194] Registered new tserver with Master: 4f097e9ce36c42e4a35c7af123779a87 (127.0.98.2:45501)
I20260619 20:55:54.379526 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40077
W20260619 20:55:54.458657 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.458842 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.458863 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.460003 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.460039 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.460063 9953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:54.461339 9953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.461501 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.461707 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.466432 9959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.466439 9961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.466434 9958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.466627 9953 server_base.cc:1034] running on GCE node
I20260619 20:55:54.466853 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.467036 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.468175 9953 hybrid_clock.cc:648] HybridClock initialized: now 1781902554468164 us; error 26 us; skew 500 ppm
I20260619 20:55:54.469394 9953 webserver.cc:460] Webserver started at http://127.0.98.3:41209/ using document root <none> and password file <none>
I20260619 20:55:54.469614 9953 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.469662 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.469810 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.470890 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "0415caa2f7c9408cb0f84dc2363496c9"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "0410808074bd5738c42561d69ac79a91"
server_key_iv: "e5675434c01cab0f6c76262f39a4bc8b"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.471269 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0415caa2f7c9408cb0f84dc2363496c9"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "0410808074bd5738c42561d69ac79a91"
server_key_iv: "e5675434c01cab0f6c76262f39a4bc8b"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.472513 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:54.473184 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.473337 9953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:54.473431 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "0415caa2f7c9408cb0f84dc2363496c9"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "0410808074bd5738c42561d69ac79a91"
server_key_iv: "e5675434c01cab0f6c76262f39a4bc8b"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.473510 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.489974 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.490300 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.490439 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.490648 9953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:54.490940 9953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:54.490993 9953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.491030 9953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:54.491060 9953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.496779 9953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37961
I20260619 20:55:54.496826 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37961 every 8 connection(s)
I20260619 20:55:54.497113 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:54.501685 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.497744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54995 (local address 127.0.98.62:33075)
0619 20:55:54.497813 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.497817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.498029 (+ 212us) server_negotiation.cc:407] Connection header received
0619 20:55:54.498505 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.498506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.498515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.498568 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.499185 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.499647 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.500531 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.500711 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.500759 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.501457 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.501458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.501458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.501462 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.501474 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.501477 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.501519 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.501600 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.501629 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:54.501892 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.497736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.3:54995)
0619 20:55:54.497929 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:54.497960 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:54.498421 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:54.498633 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:54.498636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:54.498644 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:54.499109 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.499113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.499720 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.499720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.500459 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:54.500460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:54.500779 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:54.500780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:54.500884 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:54.501333 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:54.501333 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:54.501358 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:54.501379 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:54.501531 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:54.501534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:54.501536 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:54.501586 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:55:54.501667 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260619 20:55:54.502455 10081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260619 20:55:54.502554 10081 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:54.502728 10081 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260619 20:55:54.503103 9631 ts_manager.cc:194] Registered new tserver with Master: 0415caa2f7c9408cb0f84dc2363496c9 (127.0.98.3:37961)
I20260619 20:55:54.503118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 9953
I20260619 20:55:54.503211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:54.503403 392 external_mini_cluster.cc:1428] Setting key 2e3aaaaa5e977d12ee0f4bfcb0edb0bb
I20260619 20:55:54.503549 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54995
I20260619 20:55:54.504170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:55:54.505934 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:54.505365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.62:33075)
0619 20:55:54.505398 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.505400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.505697 (+ 297us) server_negotiation.cc:407] Connection header received
0619 20:55:54.505739 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.505740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.505748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.505801 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.505892 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58672: BlockingRecv error: recv got EOF from 127.0.0.1:58672 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:55:54.506006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9691
I20260619 20:55:54.510958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9822
I20260619 20:55:54.514991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9953
I20260619 20:55:54.519008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 9600
2026-06-19T20:55:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (551 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T20:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:54Z Disabled control of system clock
I20260619 20:55:54.545027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40947
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:55:54.622144 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.622453 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.622489 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.623624 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:54.623694 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.623718 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.623745 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:54.623770 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:54.625136 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.625399 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.625689 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.630278 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.630285 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.630468 10092 server_base.cc:1034] running on GCE node
W20260619 20:55:54.630286 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.630774 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.631037 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.632187 10092 hybrid_clock.cc:648] HybridClock initialized: now 1781902554632174 us; error 28 us; skew 500 ppm
I20260619 20:55:54.633391 10092 webserver.cc:460] Webserver started at http://127.0.98.62:39411/ using document root <none> and password file <none>
I20260619 20:55:54.633630 10092 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.633677 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.633846 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.634883 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "08b284c4c3454bceb25340d7de159e57"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "d8001bfcd257a5d3f5b02b1788b18170"
server_key_iv: "bc4e2495dfdf0709102d888fcc0d7521"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.635221 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "08b284c4c3454bceb25340d7de159e57"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "d8001bfcd257a5d3f5b02b1788b18170"
server_key_iv: "bc4e2495dfdf0709102d888fcc0d7521"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.636408 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:54.637055 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.637245 10092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:54.637331 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "08b284c4c3454bceb25340d7de159e57"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "d8001bfcd257a5d3f5b02b1788b18170"
server_key_iv: "bc4e2495dfdf0709102d888fcc0d7521"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.637399 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.640607 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.640805 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.640892 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.644577 10092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40947
I20260619 20:55:54.644606 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40947 every 8 connection(s)
I20260619 20:55:54.644907 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:54.645551 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:54.648164 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: Bootstrap starting.
I20260619 20:55:54.648703 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:54.648937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10092
I20260619 20:55:54.648995 10159 log.cc:824] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:54.649044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:54.649291 392 external_mini_cluster.cc:1428] Setting key f22a31d6f87d8ff9df9a013da29bab5a
I20260619 20:55:54.650269 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: No bootstrap required, opened a new log
I20260619 20:55:54.651948 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.649850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.0.98.62:40947)
0619 20:55:54.650141 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.650149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.650179 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:55:54.650663 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.650669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.650704 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.650867 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.651524 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.651527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.651528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.651533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.651581 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.651591 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.651692 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.651723 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.651777 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":71,"threads_started":1}
I20260619 20:55:54.651979 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } }
I20260619 20:55:54.652172 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:54.652211 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b284c4c3454bceb25340d7de159e57, State: Initialized, Role: FOLLOWER
I20260619 20:55:54.652359 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } }
I20260619 20:55:54.652431 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:54.652452 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:54.652521 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:54.653324 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } }
I20260619 20:55:54.653431 10159 leader_election.cc:302] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08b284c4c3454bceb25340d7de159e57; no voters:
I20260619 20:55:54.653594 10159 leader_election.cc:288] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:54.653671 10164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:54.653805 10164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [term 1 LEADER]: Becoming Leader. State: Replica: 08b284c4c3454bceb25340d7de159e57, State: Running, Role: LEADER
I20260619 20:55:54.653868 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:54.653946 10164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } }
I20260619 20:55:54.654270 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b284c4c3454bceb25340d7de159e57. Latest consensus state: current_term: 1 leader_uuid: "08b284c4c3454bceb25340d7de159e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } } }
I20260619 20:55:54.654363 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:54.654264 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08b284c4c3454bceb25340d7de159e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08b284c4c3454bceb25340d7de159e57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40947 } } }
I20260619 20:55:54.654527 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57 [sys.catalog]: This master's current role is: LEADER
W20260619 20:55:54.655478 10180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:55:54.655553 10180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:55:54.655639 10172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:54.655798 10172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:55:54.657101 10172 catalog_manager.cc:1347] Generated new cluster ID: 7c5cebd0f8f5423a96e5c550aa153746
I20260619 20:55:54.657161 10172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:54.679970 10172 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:54.680171 10172 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:54.693969 10172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08b284c4c3454bceb25340d7de159e57: Generated new TSK 0
I20260619 20:55:54.694198 10172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:55:54.703447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:55:54.785320 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.785532 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.785554 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.786716 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.786757 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.786782 10183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:54.788192 10183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.788432 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.788733 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.793812 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.793812 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.793828 10183 server_base.cc:1034] running on GCE node
W20260619 20:55:54.793790 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.794171 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.794466 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.795609 10183 hybrid_clock.cc:648] HybridClock initialized: now 1781902554795591 us; error 45 us; skew 500 ppm
I20260619 20:55:54.796840 10183 webserver.cc:460] Webserver started at http://127.0.98.1:36657/ using document root <none> and password file <none>
I20260619 20:55:54.797075 10183 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.797122 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.797283 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.798399 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b5b976d58804bd0817a32bbc82a9b74"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "77ef3bdbc95b8375b1ad65d59696dc64"
server_key_iv: "f9f7a0a9a0a94a897e53d6e5abb456b2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.798751 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b5b976d58804bd0817a32bbc82a9b74"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "77ef3bdbc95b8375b1ad65d59696dc64"
server_key_iv: "f9f7a0a9a0a94a897e53d6e5abb456b2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.799988 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:54.800649 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.800822 10183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:54.800904 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "9b5b976d58804bd0817a32bbc82a9b74"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "77ef3bdbc95b8375b1ad65d59696dc64"
server_key_iv: "f9f7a0a9a0a94a897e53d6e5abb456b2"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.800956 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.804136 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.804339 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.804426 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.804566 10183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:54.804807 10183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:54.804836 10183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.804855 10183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:54.804868 10183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.810388 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35197
I20260619 20:55:54.810470 10310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35197 every 8 connection(s)
I20260619 20:55:54.810779 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:54.813519 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.811518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50895 (local address 127.0.98.62:40947)
0619 20:55:54.811579 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.811584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.811797 (+ 213us) server_negotiation.cc:407] Connection header received
0619 20:55:54.812226 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.812227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.812232 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.812291 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.813342 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.813343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.813344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.813348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.813361 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.813365 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.813409 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.813456 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.813468 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:55:54.813736 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.811485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40947 (local address 127.0.98.1:50895)
0619 20:55:54.811664 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:54.811703 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:54.812146 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:54.812341 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:54.812344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:54.812351 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:54.813200 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:54.813200 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:54.813227 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:54.813256 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:54.813408 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:54.813410 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:54.813413 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:54.813462 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:55:54.813507 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260619 20:55:54.814266 10311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40947
I20260619 20:55:54.814375 10311 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:54.814517 10311 heartbeater.cc:511] Master 127.0.98.62:40947 requested a full tablet report, sending...
I20260619 20:55:54.815008 10123 ts_manager.cc:194] Registered new tserver with Master: 9b5b976d58804bd0817a32bbc82a9b74 (127.0.98.1:35197)
I20260619 20:55:54.818153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10183
I20260619 20:55:54.818284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:55:54.818446 392 external_mini_cluster.cc:1428] Setting key 5dc511f1e371a95f9b874fffbcbcf64e
I20260619 20:55:54.819247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:55:54.899150 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:54.899376 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:54.899415 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:54.900810 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:54.900872 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:54.900913 10314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:54.902626 10314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:54.902846 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:54.903129 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:54.908129 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.908107 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:54.908118 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:54.908245 10314 server_base.cc:1034] running on GCE node
I20260619 20:55:54.908499 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:54.908690 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:54.909825 10314 hybrid_clock.cc:648] HybridClock initialized: now 1781902554909812 us; error 29 us; skew 500 ppm
I20260619 20:55:54.911123 10314 webserver.cc:460] Webserver started at http://127.0.98.2:44319/ using document root <none> and password file <none>
I20260619 20:55:54.911347 10314 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:54.911394 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:54.911556 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:54.912571 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "05a98332119145afaf840218f05d50c5"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "f956be4c8ff2bda5ed2a14818c3fedfc"
server_key_iv: "f3eac6017b027d644af72bfea5b0ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.912914 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05a98332119145afaf840218f05d50c5"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "f956be4c8ff2bda5ed2a14818c3fedfc"
server_key_iv: "f3eac6017b027d644af72bfea5b0ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.914155 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:55:54.914865 10328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.915035 10314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:54.915104 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "05a98332119145afaf840218f05d50c5"
format_stamp: "Formatted at 2026-06-19 20:55:54 on dist-test-slave-8bgz"
server_key: "f956be4c8ff2bda5ed2a14818c3fedfc"
server_key_iv: "f3eac6017b027d644af72bfea5b0ac0c"
server_key_version: "encryptionkey@0"
I20260619 20:55:54.915191 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:54.918334 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:54.918567 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:54.918713 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:54.918912 10314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:54.919209 10314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:54.919240 10314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.919293 10314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:54.919322 10314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:54.924994 10314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37105
I20260619 20:55:54.925076 10441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37105 every 8 connection(s)
I20260619 20:55:54.925376 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:54.928047 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.926062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57299 (local address 127.0.98.62:40947)
0619 20:55:54.926113 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:54.926116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:54.926373 (+ 257us) server_negotiation.cc:407] Connection header received
0619 20:55:54.926784 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:54.926786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:54.926791 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:54.926854 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:54.927876 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:54.927877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:54.927877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:54.927881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:54.927895 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:54.927898 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:54.927942 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:54.927987 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:55:54.927998 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:55:54.928223 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:54.926056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40947 (local address 127.0.98.2:57299)
0619 20:55:54.926265 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:54.926295 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:54.926698 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:54.926902 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:54.926905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:54.926912 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:54.927746 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:54.927747 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:54.927772 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:54.927799 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:54.927941 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:54.927943 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:54.927945 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:55:54.927992 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:55:54.928035 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260619 20:55:54.928759 10442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40947
I20260619 20:55:54.928839 10442 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:54.928992 10442 heartbeater.cc:511] Master 127.0.98.62:40947 requested a full tablet report, sending...
I20260619 20:55:54.929353 10123 ts_manager.cc:194] Registered new tserver with Master: 05a98332119145afaf840218f05d50c5 (127.0.98.2:37105)
I20260619 20:55:54.933491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10314
I20260619 20:55:54.933602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:54.933773 392 external_mini_cluster.cc:1428] Setting key d37c9466a5d8978fc7003eaba615c7d6
I20260619 20:55:54.934646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:55:55.013279 10445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:55.013491 10445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:55.013513 10445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:55.014765 10445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:55.014804 10445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:55.014855 10445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:55.016155 10445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:55.016392 10445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:55.016708 10445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:55.021706 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:55.021682 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:55.021694 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:55.021916 10445 server_base.cc:1034] running on GCE node
I20260619 20:55:55.022339 10445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:55.022599 10445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:55.023741 10445 hybrid_clock.cc:648] HybridClock initialized: now 1781902555023729 us; error 28 us; skew 500 ppm
I20260619 20:55:55.024971 10445 webserver.cc:460] Webserver started at http://127.0.98.3:42801/ using document root <none> and password file <none>
I20260619 20:55:55.025163 10445 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:55.025205 10445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:55.025292 10445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:55.026365 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "025b3f22325f4dbfa89bad83cd39ee1d"
format_stamp: "Formatted at 2026-06-19 20:55:55 on dist-test-slave-8bgz"
server_key: "60cd2992bae6d37248fdfb4ff490ed77"
server_key_iv: "3c3345861353014e8bb8990ee96138c8"
server_key_version: "encryptionkey@0"
I20260619 20:55:55.026671 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "025b3f22325f4dbfa89bad83cd39ee1d"
format_stamp: "Formatted at 2026-06-19 20:55:55 on dist-test-slave-8bgz"
server_key: "60cd2992bae6d37248fdfb4ff490ed77"
server_key_iv: "3c3345861353014e8bb8990ee96138c8"
server_key_version: "encryptionkey@0"
I20260619 20:55:55.027854 10445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:55:55.028470 10459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:55.028712 10445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:55:55.028788 10445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "025b3f22325f4dbfa89bad83cd39ee1d"
format_stamp: "Formatted at 2026-06-19 20:55:55 on dist-test-slave-8bgz"
server_key: "60cd2992bae6d37248fdfb4ff490ed77"
server_key_iv: "3c3345861353014e8bb8990ee96138c8"
server_key_version: "encryptionkey@0"
I20260619 20:55:55.028839 10445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:55.032106 10445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:55.032310 10445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:55.032402 10445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:55.032562 10445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:55.032793 10445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:55.032821 10445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:55.032841 10445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:55.032855 10445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:55.038195 10445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39135
I20260619 20:55:55.038304 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39135 every 8 connection(s)
I20260619 20:55:55.038638 10445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:55.038908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10445
I20260619 20:55:55.039007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:55.039227 392 external_mini_cluster.cc:1428] Setting key 4ae703b890ccf95862d7d165debac75d
I20260619 20:55:55.041841 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:55.039499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45767 (local address 127.0.98.62:40947)
0619 20:55:55.039551 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:55.039554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:55.039850 (+ 296us) server_negotiation.cc:407] Connection header received
0619 20:55:55.040413 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:55.040414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:55.040420 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:55.040466 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:55.041624 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:55.041625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:55.041625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:55:55.041630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:55.041643 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:55:55.041647 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:55.041696 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:55.041760 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 20:55:55.041774 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:55:55.042040 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:55.039419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40947 (local address 127.0.98.3:45767)
0619 20:55:55.039747 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:55.039778 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:55.040310 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:55.040534 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:55.040538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:55.040547 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:55.041483 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:55:55.041483 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:55.041510 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:55:55.041537 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:55.041695 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:55.041699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:55.041702 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:55.041771 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:55:55.041827 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":88,"threads_started":1}
I20260619 20:55:55.042449 10573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40947
I20260619 20:55:55.042517 10573 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:55.042657 10573 heartbeater.cc:511] Master 127.0.98.62:40947 requested a full tablet report, sending...
I20260619 20:55:55.042997 10123 ts_manager.cc:194] Registered new tserver with Master: 025b3f22325f4dbfa89bad83cd39ee1d (127.0.98.3:39135)
I20260619 20:55:55.050829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:55:55.052493 10163 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:55:55.052019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.98.62:40947)
0619 20:55:55.052106 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:55.052108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:55.052175 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:55:55.052248 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:55.052249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:55.052254 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:55.052296 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:55.052432 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38008: BlockingRecv error: recv got EOF from 127.0.0.1:38008 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 20:55:55.052553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10183
I20260619 20:55:55.057018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10314
I20260619 20:55:55.060802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10445
I20260619 20:55:55.064738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10092
2026-06-19T20:55:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:55 dist-test-slave-8bgz krb5kdc[10583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:55 dist-test-slave-8bgz krb5kdc[10583](info): set up 2 sockets
Jun 19 20:55:55 dist-test-slave-8bgz krb5kdc[10583](info): commencing operation
krb5kdc: starting...
W20260619 20:55:56.094872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:55:56.209061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:56.287173 10596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:56.287401 10596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:56.287439 10596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:56.288584 10596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:55:56.288650 10596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:56.288674 10596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:56.288704 10596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:55:56.288728 10596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:55:56.290195 10596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:56.290515 10596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:56.290725 10596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:56.295449 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.295506 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.295508 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:56.295704 10596 server_base.cc:1034] running on GCE node
I20260619 20:55:56.295944 10596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:56.296219 10596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:56.297396 10596 hybrid_clock.cc:648] HybridClock initialized: now 1781902556297384 us; error 43 us; skew 500 ppm
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:56.299731 10596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:55:56.300143 10596 webserver.cc:460] Webserver started at http://127.0.98.62:39569/ using document root <none> and password file <none>
I20260619 20:55:56.300379 10596 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:56.300454 10596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:56.300599 10596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:56.301610 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "5627e799e7e54de097c44a3b7f7dece3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "7c013a8888b0ae526cc44e7a06726cc2"
server_key_iv: "0fa5dc941153b63b812f27b04bb6b7e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.301963 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5627e799e7e54de097c44a3b7f7dece3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "7c013a8888b0ae526cc44e7a06726cc2"
server_key_iv: "0fa5dc941153b63b812f27b04bb6b7e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.303396 10596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:56.304136 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.304296 10596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:56.304396 10596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "5627e799e7e54de097c44a3b7f7dece3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "7c013a8888b0ae526cc44e7a06726cc2"
server_key_iv: "0fa5dc941153b63b812f27b04bb6b7e5"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.304479 10596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:56.322472 10596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:56.324332 10596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:56.324497 10596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:56.328511 10596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42855
I20260619 20:55:56.328558 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42855 every 8 connection(s)
I20260619 20:55:56.328912 10596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:55:56.329720 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:56.331955 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: Bootstrap starting.
I20260619 20:55:56.332513 10664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:56.332792 10664 log.cc:824] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:56.333323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10596
I20260619 20:55:56.333433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:55:56.333556 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: No bootstrap required, opened a new log
I20260619 20:55:56.333626 392 external_mini_cluster.cc:1428] Setting key 562b10a2a29a847846ee64502c5846e8
I20260619 20:55:56.336293 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } }
I20260619 20:55:56.336421 10664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:56.336442 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5627e799e7e54de097c44a3b7f7dece3, State: Initialized, Role: FOLLOWER
I20260619 20:55:56.336570 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } }
I20260619 20:55:56.336653 10664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:55:56.336695 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:55:56.336742 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:56.337392 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } }
I20260619 20:55:56.337493 10664 leader_election.cc:302] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5627e799e7e54de097c44a3b7f7dece3; no voters:
I20260619 20:55:56.337632 10664 leader_election.cc:288] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:55:56.337656 10669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:56.337845 10669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [term 1 LEADER]: Becoming Leader. State: Replica: 5627e799e7e54de097c44a3b7f7dece3, State: Running, Role: LEADER
I20260619 20:55:56.337883 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:55:56.338553 10669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } }
I20260619 20:55:56.339110 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5627e799e7e54de097c44a3b7f7dece3. Latest consensus state: current_term: 1 leader_uuid: "5627e799e7e54de097c44a3b7f7dece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } } }
I20260619 20:55:56.339257 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:55:56.339332 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5627e799e7e54de097c44a3b7f7dece3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5627e799e7e54de097c44a3b7f7dece3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42855 } } }
I20260619 20:55:56.340474 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3 [sys.catalog]: This master's current role is: LEADER
W20260619 20:55:56.340163 10683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:55:56.340592 10683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:55:56.340659 10686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:55:56.340837 10686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:56.342469 10686 catalog_manager.cc:1347] Generated new cluster ID: 60876e5549964c15971ec257a662c852
I20260619 20:55:56.342514 10686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:55:56.345257 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.334120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.62:42855)
0619 20:55:56.334326 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.334342 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.334365 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:55:56.334861 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.334867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.334912 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.335020 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:56.335766 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.336734 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.337815 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.338483 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.338603 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.341391 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:56.341395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:56.341395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:56.341407 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:56.343195 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.343609 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.343612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.343613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.343719 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.343939 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.343940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.343940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.344236 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:56.344395 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.344911 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.345050 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":133,"threads_started":1}
I20260619 20:55:56.354967 10686 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:55:56.355701 10686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:55:56.360535 10686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: Generated new TSK 0
I20260619 20:55:56.360709 10686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:55:56.389526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:56.469038 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:56.469226 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:56.469249 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:56.470355 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:56.470391 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:56.470415 10690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:55:56.471740 10690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:56.471910 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:56.472080 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:56.476833 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.477072 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.477077 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:56.477160 10690 server_base.cc:1034] running on GCE node
I20260619 20:55:56.477484 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:56.477687 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:56.478825 10690 hybrid_clock.cc:648] HybridClock initialized: now 1781902556478814 us; error 26 us; skew 500 ppm
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:56.481191 10690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:55:56.481568 10690 webserver.cc:460] Webserver started at http://127.0.98.1:44311/ using document root <none> and password file <none>
I20260619 20:55:56.481758 10690 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:56.481798 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:56.481894 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:56.482940 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "2559a51fc233400b9f254ad816718115"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "8e6993ec3b8efe7497063055c0ac7189"
server_key_iv: "b65d71db4fb776b6b9e71520c77366c4"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.483240 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2559a51fc233400b9f254ad816718115"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "8e6993ec3b8efe7497063055c0ac7189"
server_key_iv: "b65d71db4fb776b6b9e71520c77366c4"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.484563 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:55:56.485236 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.485438 10690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:56.485525 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "2559a51fc233400b9f254ad816718115"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "8e6993ec3b8efe7497063055c0ac7189"
server_key_iv: "b65d71db4fb776b6b9e71520c77366c4"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.485585 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:56.509987 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:56.511788 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:56.511902 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:56.512072 10690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:56.512326 10690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:56.512356 10690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.512378 10690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:56.512390 10690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.517874 10690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40419
I20260619 20:55:56.517953 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40419 every 8 connection(s)
I20260619 20:55:56.518296 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:55:56.524106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10690
I20260619 20:55:56.524185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:56.524375 392 external_mini_cluster.cc:1428] Setting key a443b9c611a4d45ebd2c1a7fea865ba3
I20260619 20:55:56.527659 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.519037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35553 (local address 127.0.98.62:42855)
0619 20:55:56.519101 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.519102 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.519297 (+ 195us) server_negotiation.cc:407] Connection header received
0619 20:55:56.519808 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.519809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.519822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.519882 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:56.520817 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.521320 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.522363 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.522592 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.522677 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.524885 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:56.524887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:56.524887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:56.524893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:56.526300 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.526696 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.526697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.526697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.526759 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.527014 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.527015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.527015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.527125 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:56.527205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.527491 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.527570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:55:56.527889 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.519010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42855 (local address 127.0.98.1:35553)
0619 20:55:56.519185 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.519218 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.519700 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.519959 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.519962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.520503 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:56.520685 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.520690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.521422 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.521423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.522276 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.522278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.522685 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.522686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.522812 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.523333 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:56.523341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:56.524758 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:56.526394 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.526400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.526402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.526619 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.526827 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.526828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.526828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.526938 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.527208 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:56.527215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:56.527357 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.527459 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.527611 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260619 20:55:56.528458 10819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42855
I20260619 20:55:56.528568 10819 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:56.528759 10819 heartbeater.cc:511] Master 127.0.98.62:42855 requested a full tablet report, sending...
I20260619 20:55:56.529266 10628 ts_manager.cc:194] Registered new tserver with Master: 2559a51fc233400b9f254ad816718115 (127.0.98.1:40419)
I20260619 20:55:56.529883 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35553
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:55:56.548924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:56.626857 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:56.627068 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:56.627107 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:56.628221 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:56.628283 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:56.628325 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:55:56.629791 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:56.630050 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:56.630319 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:56.635094 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.635119 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.635118 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:56.635226 10824 server_base.cc:1034] running on GCE node
I20260619 20:55:56.635490 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:56.635684 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:56.636828 10824 hybrid_clock.cc:648] HybridClock initialized: now 1781902556636809 us; error 35 us; skew 500 ppm
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:56.639406 10824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:55:56.639890 10824 webserver.cc:460] Webserver started at http://127.0.98.2:40877/ using document root <none> and password file <none>
I20260619 20:55:56.640123 10824 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:56.640189 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:56.640323 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:56.641309 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "fbe673f3197e4ac08b561541a81938b3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "69fe34ae8e7327e43b5be8d313056f69"
server_key_iv: "d4d2d1dd145278a69488c7c4b07d2c76"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.641664 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe673f3197e4ac08b561541a81938b3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "69fe34ae8e7327e43b5be8d313056f69"
server_key_iv: "d4d2d1dd145278a69488c7c4b07d2c76"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.643029 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:56.643703 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.643890 10824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:55:56.643985 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "fbe673f3197e4ac08b561541a81938b3"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "69fe34ae8e7327e43b5be8d313056f69"
server_key_iv: "d4d2d1dd145278a69488c7c4b07d2c76"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.644066 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:56.663632 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:56.665375 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:56.665535 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:56.665745 10824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:56.666040 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:56.666095 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.666129 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:56.666157 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.671620 10824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36503
I20260619 20:55:56.671656 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36503 every 8 connection(s)
I20260619 20:55:56.671986 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:55:56.673195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10824
I20260619 20:55:56.673317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:55:56.673516 392 external_mini_cluster.cc:1428] Setting key 43d41e84a4590dce1171c2f9392f4543
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:55:56.682511 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.672732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44737 (local address 127.0.98.62:42855)
0619 20:55:56.672775 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.672776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.672987 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:55:56.673590 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.673591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.673603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.673686 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:56.674781 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.675552 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.677083 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.677264 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.677322 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.679740 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:56.679742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:56.679742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:56.679749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:56.681113 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.681521 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.681522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.681523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.681581 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.681854 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.681855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.681855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.681966 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:56.682044 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.682341 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.682422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:55:56.682690 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.672712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42855 (local address 127.0.98.2:44737)
0619 20:55:56.672876 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.672908 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.673490 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.673779 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.673784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.674507 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:56.674687 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.674692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.675651 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.675652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.677002 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.677005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.677344 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.677345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.677500 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.678045 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:56.678057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:56.679646 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:56.681198 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.681204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.681206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.681432 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.681656 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.681657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.681657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.681761 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.682034 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:56.682039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:56.682179 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.682311 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.682466 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260619 20:55:56.683214 10953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42855
I20260619 20:55:56.683291 10953 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:56.683506 10953 heartbeater.cc:511] Master 127.0.98.62:42855 requested a full tablet report, sending...
I20260619 20:55:56.683847 10628 ts_manager.cc:194] Registered new tserver with Master: fbe673f3197e4ac08b561541a81938b3 (127.0.98.2:36503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 20:55:56.684350 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:55:56.698663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:55:56.778034 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:55:56.778271 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:55:56.778311 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:55:56.779457 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:55:56.779539 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:55:56.779582 10958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:55:56.780970 10958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:55:56.781220 10958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:55:56.781446 10958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:55:56.786288 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.786301 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:55:56.786301 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:55:56.786343 10958 server_base.cc:1034] running on GCE node
I20260619 20:55:56.786625 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:55:56.786823 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:55:56.787997 10958 hybrid_clock.cc:648] HybridClock initialized: now 1781902556787985 us; error 28 us; skew 500 ppm
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:55:56.790486 10958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:55:56.790944 10958 webserver.cc:460] Webserver started at http://127.0.98.3:46737/ using document root <none> and password file <none>
I20260619 20:55:56.791157 10958 fs_manager.cc:359] Metadata directory not provided
I20260619 20:55:56.791224 10958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:55:56.791376 10958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:55:56.792392 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "8da6a85783ab49a295bbd5f59703c2c0"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "414008008bec8d7f1a4194ca96744b09"
server_key_iv: "9377bd69dec03ab324a65f6eac885cf1"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.792750 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8da6a85783ab49a295bbd5f59703c2c0"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "414008008bec8d7f1a4194ca96744b09"
server_key_iv: "9377bd69dec03ab324a65f6eac885cf1"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.794014 10958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:55:56.794796 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.794972 10958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.795068 10958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "8da6a85783ab49a295bbd5f59703c2c0"
format_stamp: "Formatted at 2026-06-19 20:55:56 on dist-test-slave-8bgz"
server_key: "414008008bec8d7f1a4194ca96744b09"
server_key_iv: "9377bd69dec03ab324a65f6eac885cf1"
server_key_version: "encryptionkey@0"
I20260619 20:55:56.795153 10958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:55:56.810470 10958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:55:56.812201 10958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:55:56.812366 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:55:56.812594 10958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:55:56.812906 10958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:55:56.812960 10958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.813000 10958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:55:56.813030 10958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:55:56.818938 10958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41405
I20260619 20:55:56.818995 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41405 every 8 connection(s)
I20260619 20:55:56.819335 10958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:55:56.823231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 10958
I20260619 20:55:56.823330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:55:56.823504 392 external_mini_cluster.cc:1428] Setting key 6b6a222aa1c6a755306bbee0bc5e6123
Jun 19 20:55:56 dist-test-slave-8bgz krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:55:56.828354 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.820141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57057 (local address 127.0.98.62:42855)
0619 20:55:56.820220 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.820222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.820408 (+ 186us) server_negotiation.cc:407] Connection header received
0619 20:55:56.820852 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.820853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.820863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.820909 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:56.821835 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.822438 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.823671 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.823991 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.824068 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.826109 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:56.826110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:56.826110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:56.826116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:56.827299 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.827636 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.827636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.827637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.827684 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.827881 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.827881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.827882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.827975 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:56.828023 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.828235 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.828292 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:55:56.828512 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.820103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42855 (local address 127.0.98.3:57057)
0619 20:55:56.820286 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.820319 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.820770 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.820982 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.820985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.821547 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:55:56.821726 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.821731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.822536 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.822537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.823581 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.823584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.824079 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.824080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.824211 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.824739 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:55:56.824755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:55:56.826034 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:55:56.827367 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.827373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.827374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.827573 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.827739 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:55:56.827739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:55:56.827740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:55:56.827829 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:55:56.828027 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:55:56.828031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:55:56.828134 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.828210 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.828325 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260619 20:55:56.829002 11087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42855
I20260619 20:55:56.829077 11087 heartbeater.cc:464] Registering TS with master...
I20260619 20:55:56.829201 11087 heartbeater.cc:511] Master 127.0.98.62:42855 requested a full tablet report, sending...
I20260619 20:55:56.829551 10628 ts_manager.cc:194] Registered new tserver with Master: 8da6a85783ab49a295bbd5f59703c2c0 (127.0.98.3:41405)
I20260619 20:55:56.829869 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57057
I20260619 20:55:56.835132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:55:56.842056 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.836365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.98.62:42855)
0619 20:55:56.836406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.836407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.836481 (+ 74us) server_negotiation.cc:407] Connection header received
0619 20:55:56.836519 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.836520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.836528 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.836573 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:55:56.837098 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.837655 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.838480 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.838728 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.838781 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.839974 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:55:56.839975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:55:56.839976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:55:56.839983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:55:56.841164 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.841465 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.841466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.841466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.841513 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:55:56.841639 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:55:56.841639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:55:56.841639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:55:56.841760 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:55:56.841842 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.841942 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.841996 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:55:56.843502 10628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:55:56.844985 10628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:55:56.851219 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36503 (local address 127.0.0.1:33006)
0619 20:55:56.847427 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.847439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.847488 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.847864 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.847867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.847870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:56.847933 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.847934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.848912 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.848913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.850104 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.850106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.850899 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.850900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.850987 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.850999 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.851095 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.851151 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":44,"threads_started":1}
I20260619 20:55:56.851243 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.0.98.2:36503)
0619 20:55:56.847705 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.847710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.847720 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:55:56.847760 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.847762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.847777 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.847861 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:56.848021 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.848789 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.850196 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.850810 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.850888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.850948 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.851130 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.851168 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260619 20:55:56.851699 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41405 (local address 127.0.0.1:34832)
0619 20:55:56.847556 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.847567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.847613 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.848088 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.848089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.848091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:56.848128 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.848129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.849357 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.849358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.850377 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.850379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.851439 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.851440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.851525 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.851531 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.851580 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.851620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":36,"threads_started":1}
I20260619 20:55:56.851900 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40419 (local address 127.0.0.1:42616)
0619 20:55:56.847245 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.847295 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.847358 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.847900 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.847901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.847903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:56.847954 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.847956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.849332 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.849333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.850402 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.850404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.851572 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.851572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.851626 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.851629 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.851658 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.851691 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260619 20:55:56.852031 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.0.98.1:40419)
0619 20:55:56.847728 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.847733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.847742 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:55:56.847776 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.847788 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.847804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.848029 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:56.848102 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.849216 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.850505 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.851200 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.851764 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.851851 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.851882 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.851930 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":74,"threads_started":1}
I20260619 20:55:56.852052 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 4c125b4f8fe4459b87948df551c15a08 (DEFAULT_TABLE table=test-table [id=b8d54e264c6c48b6a9091a81aa087546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:56.852406 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.3:41405)
0619 20:55:56.847651 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.847655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.847663 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:55:56.847732 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.847734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.847751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.848021 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:56.848204 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.849247 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.850469 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.851194 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.851266 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.851386 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.852060 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.852328 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260619 20:55:56.852464 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c125b4f8fe4459b87948df551c15a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:56.852680 10753 tablet_service.cc:1467] Processing CreateTablet for tablet 4c125b4f8fe4459b87948df551c15a08 (DEFAULT_TABLE table=test-table [id=b8d54e264c6c48b6a9091a81aa087546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:56.853008 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c125b4f8fe4459b87948df551c15a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:56.852942 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 4c125b4f8fe4459b87948df551c15a08 (DEFAULT_TABLE table=test-table [id=b8d54e264c6c48b6a9091a81aa087546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:55:56.853210 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c125b4f8fe4459b87948df551c15a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:55:56.855235 11101 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Bootstrap starting.
I20260619 20:55:56.855857 11101 tablet_bootstrap.cc:654] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:56.856294 11102 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Bootstrap starting.
I20260619 20:55:56.856434 11101 log.cc:824] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:56.856941 11102 tablet_bootstrap.cc:654] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:56.857190 11101 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: No bootstrap required, opened a new log
I20260619 20:55:56.857268 11101 ts_tablet_manager.cc:1397] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:55:56.857501 11102 log.cc:824] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:56.857812 11103 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Bootstrap starting.
I20260619 20:55:56.858155 11102 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: No bootstrap required, opened a new log
I20260619 20:55:56.858258 11102 ts_tablet_manager.cc:1397] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:56.858450 11103 tablet_bootstrap.cc:654] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Neither blocks nor log segments found. Creating new log.
I20260619 20:55:56.858951 11103 log.cc:824] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Log is configured to *not* fsync() on all Append() calls
I20260619 20:55:56.859398 11101 raft_consensus.cc:348] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.859519 11101 raft_consensus.cc:374] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:56.859539 11101 raft_consensus.cc:729] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe673f3197e4ac08b561541a81938b3, State: Initialized, Role: FOLLOWER
I20260619 20:55:56.859669 11101 consensus_queue.cc:260] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.859944 11101 ts_tablet_manager.cc:1428] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:55:56.859972 10953 heartbeater.cc:503] Master 127.0.98.62:42855 was elected leader, sending a full tablet report...
I20260619 20:55:56.859889 11102 raft_consensus.cc:348] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.860045 11102 raft_consensus.cc:374] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:56.860080 11102 raft_consensus.cc:729] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da6a85783ab49a295bbd5f59703c2c0, State: Initialized, Role: FOLLOWER
I20260619 20:55:56.860196 11102 consensus_queue.cc:260] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.860478 11102 ts_tablet_manager.cc:1428] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:55:56.860478 11087 heartbeater.cc:503] Master 127.0.98.62:42855 was elected leader, sending a full tablet report...
I20260619 20:55:56.860558 11103 tablet_bootstrap.cc:492] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: No bootstrap required, opened a new log
I20260619 20:55:56.860617 11103 ts_tablet_manager.cc:1397] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:55:56.862195 11103 raft_consensus.cc:348] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.862382 11103 raft_consensus.cc:374] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:55:56.862406 11103 raft_consensus.cc:729] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2559a51fc233400b9f254ad816718115, State: Initialized, Role: FOLLOWER
I20260619 20:55:56.862506 11103 consensus_queue.cc:260] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.862762 11103 ts_tablet_manager.cc:1428] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:55:56.862777 10819 heartbeater.cc:503] Master 127.0.98.62:42855 was elected leader, sending a full tablet report...
W20260619 20:55:56.922652 10954 tablet.cc:2369] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:56.986734 11108 raft_consensus.cc:482] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:55:56.986845 11108 raft_consensus.cc:504] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.987144 11108 leader_election.cc:288] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2559a51fc233400b9f254ad816718115 (127.0.98.1:40419), fbe673f3197e4ac08b561541a81938b3 (127.0.98.2:36503)
I20260619 20:55:56.990077 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.987339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36503 (local address 127.0.98.3:52741)
0619 20:55:56.987400 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.987413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.987459 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.987638 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.987639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.987641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:56.987714 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.987716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.988397 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.988398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.989251 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.989254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.989816 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.989899 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.989902 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.989957 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.989987 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:55:56.990101 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.987407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52741 (local address 127.0.98.2:36503)
0619 20:55:56.987450 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.987453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.987464 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:55:56.987530 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.987531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.987544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.987621 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:56.987787 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.988321 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989345 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.989739 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989792 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.989800 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.989984 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.990030 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:55:56.990176 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.987406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60203 (local address 127.0.98.1:40419)
0619 20:55:56.987450 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:56.987453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:56.987625 (+ 172us) server_negotiation.cc:407] Connection header received
0619 20:55:56.987655 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:56.987656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:56.987668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:56.987728 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:55:56.987933 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.988408 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989450 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.989825 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989894 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.989903 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:56.990065 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 20:55:56.990107 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:55:56.990411 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:56.987344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40419 (local address 127.0.98.3:60203)
0619 20:55:56.987537 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:56.987548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:55:56.987610 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:55:56.987735 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:55:56.987736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:55:56.987737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:55:56.987838 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.987839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.988520 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.988521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.989371 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:55:56.989373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:55:56.989901 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:55:56.989901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:55:56.989983 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:56.989987 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:55:56.990041 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:55:56.990077 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":12,"thread_start_us":98,"threads_started":1}
I20260619 20:55:56.990479 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c125b4f8fe4459b87948df551c15a08" candidate_uuid: "8da6a85783ab49a295bbd5f59703c2c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe673f3197e4ac08b561541a81938b3" is_pre_election: true
I20260619 20:55:56.990639 10907 raft_consensus.cc:2454] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da6a85783ab49a295bbd5f59703c2c0 in term 0.
I20260619 20:55:56.990726 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c125b4f8fe4459b87948df551c15a08" candidate_uuid: "8da6a85783ab49a295bbd5f59703c2c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2559a51fc233400b9f254ad816718115" is_pre_election: true
I20260619 20:55:56.990892 10976 leader_election.cc:302] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da6a85783ab49a295bbd5f59703c2c0, fbe673f3197e4ac08b561541a81938b3; no voters:
I20260619 20:55:56.990929 10773 raft_consensus.cc:2454] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da6a85783ab49a295bbd5f59703c2c0 in term 0.
I20260619 20:55:56.991008 11108 raft_consensus.cc:2781] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:55:56.991076 11108 raft_consensus.cc:482] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:55:56.991175 11108 raft_consensus.cc:3037] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:56.991828 11108 raft_consensus.cc:504] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.991945 11108 leader_election.cc:288] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [CANDIDATE]: Term 1 election: Requested vote from peers 2559a51fc233400b9f254ad816718115 (127.0.98.1:40419), fbe673f3197e4ac08b561541a81938b3 (127.0.98.2:36503)
I20260619 20:55:56.992079 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c125b4f8fe4459b87948df551c15a08" candidate_uuid: "8da6a85783ab49a295bbd5f59703c2c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2559a51fc233400b9f254ad816718115"
I20260619 20:55:56.992100 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c125b4f8fe4459b87948df551c15a08" candidate_uuid: "8da6a85783ab49a295bbd5f59703c2c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe673f3197e4ac08b561541a81938b3"
I20260619 20:55:56.992158 10907 raft_consensus.cc:3037] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:56.992156 10773 raft_consensus.cc:3037] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:55:56.993045 10907 raft_consensus.cc:2454] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da6a85783ab49a295bbd5f59703c2c0 in term 1.
I20260619 20:55:56.993206 10976 leader_election.cc:302] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da6a85783ab49a295bbd5f59703c2c0, fbe673f3197e4ac08b561541a81938b3; no voters:
I20260619 20:55:56.993323 10773 raft_consensus.cc:2454] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da6a85783ab49a295bbd5f59703c2c0 in term 1.
I20260619 20:55:56.993429 11108 raft_consensus.cc:2781] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:55:56.993599 11108 raft_consensus.cc:686] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8da6a85783ab49a295bbd5f59703c2c0, State: Running, Role: LEADER
I20260619 20:55:56.993695 11108 consensus_queue.cc:237] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } }
I20260619 20:55:56.994537 10627 catalog_manager.cc:5495] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8da6a85783ab49a295bbd5f59703c2c0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8da6a85783ab49a295bbd5f59703c2c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da6a85783ab49a295bbd5f59703c2c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 } health_report { overall_health: UNKNOWN } } }
W20260619 20:55:57.019011 10820 tablet.cc:2369] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:55:57.034978 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:57.032527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.0.98.3:41405)
0619 20:55:57.032593 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:55:57.032596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:55:57.032621 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:55:57.032667 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:55:57.032668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:55:57.032687 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:55:57.032754 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:55:57.032917 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:57.033382 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:57.034145 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:55:57.034400 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:55:57.034468 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:55:57.034632 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:55:57.034744 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:55:57.034839 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:55:57.034857 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 20:55:57.034877 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:55:57.036815 10773 raft_consensus.cc:1261] T 4c125b4f8fe4459b87948df551c15a08 P 2559a51fc233400b9f254ad816718115 [term 1 FOLLOWER]: Refusing update from remote peer 8da6a85783ab49a295bbd5f59703c2c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:57.036823 10907 raft_consensus.cc:1261] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 1 FOLLOWER]: Refusing update from remote peer 8da6a85783ab49a295bbd5f59703c2c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:55:57.037210 11108 consensus_queue.cc:1035] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe673f3197e4ac08b561541a81938b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:57.037388 11111 consensus_queue.cc:1035] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2559a51fc233400b9f254ad816718115" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:55:57.039676 11114 mvcc.cc:205] Tried to move back new op lower bound from 7298672873620365312 to 7298672873447104512. Current Snapshot: MvccSnapshot[applied={T|T < 7298672873620365312}]
I20260619 20:55:57.042052 10627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:57.042131 10627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:55:57.042877 10627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5627e799e7e54de097c44a3b7f7dece3: Sending DeleteTablet for 3 replicas of tablet 4c125b4f8fe4459b87948df551c15a08
I20260619 20:55:57.043216 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c125b4f8fe4459b87948df551c15a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33006
I20260619 20:55:57.043241 10753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c125b4f8fe4459b87948df551c15a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42616
I20260619 20:55:57.043445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10690
I20260619 20:55:57.043521 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c125b4f8fe4459b87948df551c15a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:55:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34832
I20260619 20:55:57.043548 11123 tablet_replica.cc:330] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: stopping tablet replica
I20260619 20:55:57.043663 11123 raft_consensus.cc:2229] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:55:57.043771 11123 raft_consensus.cc:2258] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:57.044107 11125 tablet_replica.cc:330] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: stopping tablet replica
I20260619 20:55:57.044198 11125 raft_consensus.cc:2229] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:55:57.044291 11125 raft_consensus.cc:2258] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:55:57.044620 11123 ts_tablet_manager.cc:1905] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:57.044726 11125 ts_tablet_manager.cc:1905] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:55:57.046494 11125 ts_tablet_manager.cc:1918] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:57.046579 11125 log.cc:1196] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/4c125b4f8fe4459b87948df551c15a08
I20260619 20:55:57.046717 11123 ts_tablet_manager.cc:1918] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:55:57.046794 11123 log.cc:1196] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/4c125b4f8fe4459b87948df551c15a08
I20260619 20:55:57.046875 11125 ts_tablet_manager.cc:1939] T 4c125b4f8fe4459b87948df551c15a08 P 8da6a85783ab49a295bbd5f59703c2c0: Deleting consensus metadata
I20260619 20:55:57.047242 10612 catalog_manager.cc:4841] TS 8da6a85783ab49a295bbd5f59703c2c0 (127.0.98.3:41405): tablet 4c125b4f8fe4459b87948df551c15a08 (table test-table [id=b8d54e264c6c48b6a9091a81aa087546]) successfully deleted
I20260619 20:55:57.047385 11123 ts_tablet_manager.cc:1939] T 4c125b4f8fe4459b87948df551c15a08 P fbe673f3197e4ac08b561541a81938b3: Deleting consensus metadata
I20260619 20:55:57.047837 10614 catalog_manager.cc:4841] TS fbe673f3197e4ac08b561541a81938b3 (127.0.98.2:36503): tablet 4c125b4f8fe4459b87948df551c15a08 (table test-table [id=b8d54e264c6c48b6a9091a81aa087546]) successfully deleted
W20260619 20:55:57.050474 10612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40419 (error 108)
I20260619 20:55:57.050693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10824
I20260619 20:55:57.054657 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:55:57.054505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40419 (local address 127.0.0.1:42628)
0619 20:55:57.054573 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:55:57.054615 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 20:55:57.054805 10612 catalog_manager.cc:4584] TS 2559a51fc233400b9f254ad816718115 (127.0.98.1:40419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c125b4f8fe4459b87948df551c15a08: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40419: connect: Connection refused (error 111)
I20260619 20:55:57.055927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10958
I20260619 20:55:57.060170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 10596
2026-06-19T20:55:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1995 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:55:57 dist-test-slave-8bgz krb5kdc[11129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:55:57 dist-test-slave-8bgz krb5kdc[11129](info): set up 2 sockets
Jun 19 20:55:57 dist-test-slave-8bgz krb5kdc[11129](info): commencing operation
krb5kdc: starting...
W20260619 20:55:58.095252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:55:58 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:55:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 20:55:58.248960 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_bKm47/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/postgres -l logfile start
2026-06-19 20:56:00.624 UTC [11159] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:56:00.624 UTC [11159] LOG: listening on IPv4 address "127.0.98.20", port 60517
2026-06-19 20:56:00.628 UTC [11159] LOG: listening on Unix socket "/tmp/.s.PGSQL.60517"
2026-06-19 20:56:00.662 UTC [11162] LOG: database system was shut down at 2026-06-19 20:55:59 UTC
2026-06-19 20:56:00.666 UTC [11159] LOG: database system is ready to accept connections
I20260619 20:56:01.618880 392 mini_postgres.cc:96] Postgres bound to 60517
2026-06-19 20:56:01.624 UTC [11170] FATAL: database "slave" does not exist
127.0.98.20:60517 - accepting connections
I20260619 20:56:01.625401 392 mini_ranger.cc:161] Starting Ranger...
I20260619 20:56:01.639818 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:56:02.104916 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:56:02.105006 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin
2026-06-19 20:56:02,377 [I] DB FLAVOR :POSTGRES
2026-06-19 20:56:02,377 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:56:02,377 [I] Checking connection..
2026-06-19 20:56:02,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:56:02,626 [I] Checking connection passed.
2026-06-19 20:56:02,626 [I] --------- Verifying version history table ---------
2026-06-19 20:56:02,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:02,883 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:56:02,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:03,107 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:56:03,107 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:56:03,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:56:03.361 UTC [11234] WARNING: there is no transaction in progress
2026-06-19 20:56:03,367 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:56:03,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:03,609 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:56:03,609 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:56:03,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:56:03,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:56:04,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8bgz.c.gcp-upstream.internal','N') ;"
2026-06-19 20:56:04,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:56:04,554 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:56:04,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:56:04,814 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:56:04,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:56:05,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:56:05,306 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:56:05,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:56:06.215 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.231 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.247 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.263 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.278 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.298 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.351 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.357 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.371 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.381 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.392 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.404 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.414 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.420 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.428 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:06.434 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 20:56:07,033 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:56:07,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:56:07,264 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:56:07,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:56:07,512 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:56:07,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:56:07,738 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:56:07,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:56:07,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:56:08,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8bgz.c.gcp-upstream.internal';"
2026-06-19 20:56:08,467 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:56:08,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:56:08,712 [I] DB_PATCHES have already been applied
I20260619 20:56:08.718065 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_bKm47/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:56:08.718142 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 20:56:08.720943 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44967
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:56:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:56:09.827052 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.202s user 0.003s sys 0.006s
Jun 19, 2026 8:56:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:56:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 8:56:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:56:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:56:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:56:30.206944 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:56:30 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902590, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:56:30.286366 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:56:30.295382 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:56:31.122038 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:56:31.122120 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms
I20260619 20:56:31.122140 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60517
2026-06-19 20:56:31,295 [I] DB FLAVOR :POSTGRES
2026-06-19 20:56:31,295 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:56:31,295 [I] Checking connection
2026-06-19 20:56:31,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:56:31,521 [I] connection success
2026-06-19 20:56:31,522 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:56:31,522 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:56:31,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:56:31,755 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:56:31,755 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:56:31,755 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:56:31,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:56:32,020 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:56:32.441582 392 mini_ranger_kms.cc:310] Created kms service
I20260619 20:56:32.637879 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:56:32.678344 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:56:32 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902590, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:33.038316 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:56:33.038589 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:56:33.038616 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 20:56:33.042301 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11522
I20260619 20:56:33.042423 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39393
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:56:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 20:56:35.278393 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.992s user 0.006s sys 0.150s
I20260619 20:56:35.278535 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:56:35.278621 392 mini_ranger_kms.cc:401] 127.0.98.20:39393/kms/v1/keys
Jun 19 20:56:35 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902595, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 20:56:35 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902595, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:56:36 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902595, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:56:38 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:56:38.434530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43949
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39393/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:56:38.534492 11573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:56:38.534703 11573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:56:38.534729 11573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:56:38.535832 11573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:56:38.535867 11573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:56:38.535881 11573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:56:38.535893 11573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:56:38.535905 11573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:56:38.537335 11573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39393/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43949
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:56:38.537508 11573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:56:38.537668 11573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:56:38.542425 11578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:38.542483 11579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:38.542430 11581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:56:38.542476 11573 server_base.cc:1034] running on GCE node
I20260619 20:56:38.542901 11573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:56:38.543211 11573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:56:38.544361 11573 hybrid_clock.cc:648] HybridClock initialized: now 1781902598544359 us; error 38 us; skew 500 ppm
Jun 19 20:56:38 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:56:38.549543 11573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:56:38.549947 11573 webserver.cc:460] Webserver started at http://127.0.98.62:36759/ using document root <none> and password file <none>
I20260619 20:56:38.550159 11573 fs_manager.cc:359] Metadata directory not provided
I20260619 20:56:38.550237 11573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:56:38.550396 11573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:56:38 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:38.601989 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "aec9f2df610e45f29ff9e4424a055e70"
format_stamp: "Formatted at 2026-06-19 20:56:38 on dist-test-slave-8bgz"
server_key: "62d0af99cd882c71a5155ce67e9188ce"
server_key_iv: "05da12e5293db2058de7ee88761ee51a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:38.602516 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "aec9f2df610e45f29ff9e4424a055e70"
format_stamp: "Formatted at 2026-06-19 20:56:38 on dist-test-slave-8bgz"
server_key: "62d0af99cd882c71a5155ce67e9188ce"
server_key_iv: "05da12e5293db2058de7ee88761ee51a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:38.604031 11573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:56:38.617707 11588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:38.617936 11573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:56:38.618019 11573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "aec9f2df610e45f29ff9e4424a055e70"
format_stamp: "Formatted at 2026-06-19 20:56:38 on dist-test-slave-8bgz"
server_key: "62d0af99cd882c71a5155ce67e9188ce"
server_key_iv: "05da12e5293db2058de7ee88761ee51a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:38.618095 11573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:56:38.635430 11573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:56:38.637432 11573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:56:38.637538 11573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:56:38.642191 11573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43949
I20260619 20:56:38.642251 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43949 every 8 connection(s)
I20260619 20:56:38.642678 11573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:56:38.643330 11641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:56:38.645676 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: Bootstrap starting.
I20260619 20:56:38.646346 11641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: Neither blocks nor log segments found. Creating new log.
I20260619 20:56:38.646653 11641 log.cc:824] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: Log is configured to *not* fsync() on all Append() calls
I20260619 20:56:38.647349 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: No bootstrap required, opened a new log
I20260619 20:56:38.649384 11641 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260619 20:56:38.649523 11641 raft_consensus.cc:374] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:56:38.649549 11641 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec9f2df610e45f29ff9e4424a055e70, State: Initialized, Role: FOLLOWER
I20260619 20:56:38.649732 11641 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260619 20:56:38.649802 11641 raft_consensus.cc:388] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:56:38.649827 11641 raft_consensus.cc:482] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:56:38.649856 11641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:56:38.650509 11641 raft_consensus.cc:504] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260619 20:56:38.650596 11641 leader_election.cc:302] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aec9f2df610e45f29ff9e4424a055e70; no voters:
I20260619 20:56:38.650807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 11573
I20260619 20:56:38.650830 11641 leader_election.cc:288] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:56:38.650851 11644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:56:38.650916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:56:38.650985 11644 raft_consensus.cc:686] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [term 1 LEADER]: Becoming Leader. State: Replica: aec9f2df610e45f29ff9e4424a055e70, State: Running, Role: LEADER
I20260619 20:56:38.651089 11641 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:56:38.651103 392 external_mini_cluster.cc:1428] Setting key 48fa85b3e7a2065b8f3f76cc54bba2e4
I20260619 20:56:38.651059 11644 consensus_queue.cc:237] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260619 20:56:38.651822 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aec9f2df610e45f29ff9e4424a055e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } } }
I20260619 20:56:38.651909 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [sys.catalog]: This master's current role is: LEADER
I20260619 20:56:38.652136 11646 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec9f2df610e45f29ff9e4424a055e70. Latest consensus state: current_term: 1 leader_uuid: "aec9f2df610e45f29ff9e4424a055e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec9f2df610e45f29ff9e4424a055e70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } } }
I20260619 20:56:38.652241 11646 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70 [sys.catalog]: This master's current role is: LEADER
I20260619 20:56:38.652599 11655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:56:38.652757 11641 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:56:38.653163 11655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:56:38.658675 11655 catalog_manager.cc:1347] Generated new cluster ID: 72b2a7013348484cabbf8bc3253373ca
I20260619 20:56:38.658744 11655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:56:38 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:56:38.666386 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:38.651600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.0.98.62:43949)
0619 20:56:38.651886 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:38.651896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:38.651942 (+ 46us) server_negotiation.cc:407] Connection header received
0619 20:56:38.652079 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:38.652087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:38.652146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:38.652392 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:56:38.653542 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:38.657455 (+ 3913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:38.658674 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:38.659417 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:38.659538 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:38.662953 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:56:38.662957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:56:38.662958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:56:38.662971 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:56:38.664713 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:38.665091 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:38.665094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:38.665095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:38.665204 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:38.665441 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:38.665442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:38.665442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:38.665701 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:56:38.665864 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:38.665974 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:56:38.666101 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
I20260619 20:56:38.673813 11655 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:56:38.674551 11655 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:56:38.677381 11655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: Generated new TSK 0
I20260619 20:56:38.677558 11655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:56:39.363580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:56:39.469209 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:56:39.469470 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:56:39.469506 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:56:39.471128 11686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:56:39.471184 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:56:39.471221 11686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:56:39.473031 11686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:56:39.473268 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:56:39.473515 11686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:56:39.478560 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:39.478524 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:39.478513 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:56:39.478735 11686 server_base.cc:1034] running on GCE node
I20260619 20:56:39.479125 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:56:39.479372 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:56:39.480545 11686 hybrid_clock.cc:648] HybridClock initialized: now 1781902599480520 us; error 37 us; skew 500 ppm
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:56:39.483716 11686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:56:39.484155 11686 webserver.cc:460] Webserver started at http://127.0.98.1:36815/ using document root <none> and password file <none>
I20260619 20:56:39.484396 11686 fs_manager.cc:359] Metadata directory not provided
I20260619 20:56:39.484491 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:56:39.484648 11686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:39.495831 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb2a7171e0be4a9b8bb5e6f31049f887"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "2fd6d7d477acc5d48bc90e7f7d3ffa19"
server_key_iv: "5397dd9ed916330491a99723aea93a1b"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.496261 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb2a7171e0be4a9b8bb5e6f31049f887"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "2fd6d7d477acc5d48bc90e7f7d3ffa19"
server_key_iv: "5397dd9ed916330491a99723aea93a1b"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.498900 11686 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:56:39.514762 11703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.515008 11686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:56:39.515118 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "bb2a7171e0be4a9b8bb5e6f31049f887"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "2fd6d7d477acc5d48bc90e7f7d3ffa19"
server_key_iv: "5397dd9ed916330491a99723aea93a1b"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.515221 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:56:39.543488 11686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:56:39.546106 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:56:39.546342 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:56:39.546658 11686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:56:39.547089 11686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:56:39.547186 11686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.547290 11686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:56:39.547370 11686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.555965 11686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43319
I20260619 20:56:39.556411 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:56:39.560197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 11686
I20260619 20:56:39.560302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:56:39.560480 392 external_mini_cluster.cc:1428] Setting key 05fcfdfe5d86effea1e324555715d033
I20260619 20:56:39.562326 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43319 every 8 connection(s)
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:56:39.568111 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:39.557309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43949 (local address 127.0.98.1:48915)
0619 20:56:39.557493 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:39.557542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:39.558180 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:39.558628 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:39.558632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:39.558872 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:56:39.559132 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:39.559137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.560158 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.560159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:39.561463 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:39.561465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.562019 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.562020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:39.562162 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:39.562836 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:56:39.562845 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:56:39.564450 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:56:39.566523 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:39.566529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:39.566532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:39.566821 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:39.567034 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:39.567035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:39.567035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:39.567156 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:39.567439 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:56:39.567445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:56:39.567627 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 20:56:39.567740 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 20:56:39.567884 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260619 20:56:39.568439 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:39.557609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48915 (local address 127.0.98.62:43949)
0619 20:56:39.557732 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:39.557734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:39.557746 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:56:39.558404 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:39.558405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:39.558420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:39.558486 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:56:39.559242 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.560054 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.561558 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.561892 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.561955 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:39.564546 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:56:39.564548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:56:39.564548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:56:39.564555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:56:39.566434 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:39.566905 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:39.566907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:39.566907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:39.566969 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:39.567225 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:39.567226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:39.567226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:39.567357 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:56:39.567415 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:39.568272 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 20:56:39.568348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260619 20:56:39.568908 11817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43949
I20260619 20:56:39.569000 11817 heartbeater.cc:464] Registering TS with master...
I20260619 20:56:39.569181 11817 heartbeater.cc:511] Master 127.0.98.62:43949 requested a full tablet report, sending...
I20260619 20:56:39.569711 11605 ts_manager.cc:194] Registered new tserver with Master: bb2a7171e0be4a9b8bb5e6f31049f887 (127.0.98.1:43319)
I20260619 20:56:39.570617 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48915
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:56:39.591076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:56:39.718093 11823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:56:39.718469 11823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:56:39.718549 11823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 20:56:39.720942 11823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:56:39.721030 11823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:56:39.721107 11823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:56:39.723352 11823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:56:39.723809 11823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:56:39.724084 11823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:56:39.730029 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:39.730448 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:39.734342 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:56:39.737490 11823 server_base.cc:1034] running on GCE node
I20260619 20:56:39.737735 11823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:56:39.738004 11823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:56:39.739146 11823 hybrid_clock.cc:648] HybridClock initialized: now 1781902599739125 us; error 35 us; skew 500 ppm
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:56:39.743965 11823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:56:39.744479 11823 webserver.cc:460] Webserver started at http://127.0.98.2:33037/ using document root <none> and password file <none>
I20260619 20:56:39.744735 11823 fs_manager.cc:359] Metadata directory not provided
I20260619 20:56:39.744787 11823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:56:39.744915 11823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:39.774914 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "c248719e3eb74abd8b27249147481b50"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "447b6c71675a40a95f95bf9512730347"
server_key_iv: "db81545cf542a6143842485558e3b73c"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.775449 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c248719e3eb74abd8b27249147481b50"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "447b6c71675a40a95f95bf9512730347"
server_key_iv: "db81545cf542a6143842485558e3b73c"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.782605 11823 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260619 20:56:39.801532 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.801789 11823 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260619 20:56:39.801885 11823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "c248719e3eb74abd8b27249147481b50"
format_stamp: "Formatted at 2026-06-19 20:56:39 on dist-test-slave-8bgz"
server_key: "447b6c71675a40a95f95bf9512730347"
server_key_iv: "db81545cf542a6143842485558e3b73c"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:39.801981 11823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:56:39.820436 11823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:56:39.823110 11823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:56:39.823319 11823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:56:39.823601 11823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:56:39.823968 11823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:56:39.824060 11823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.824144 11823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:56:39.824205 11823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:39.832010 11823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43787
I20260619 20:56:39.832557 11823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:56:39.834654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 11823
I20260619 20:56:39.834745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:56:39.834954 392 external_mini_cluster.cc:1428] Setting key 6e51465b4d706a8375bf95bf3859296d
I20260619 20:56:39.834394 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43787 every 8 connection(s)
Jun 19 20:56:39 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:56:39.847183 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:39.835430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44887 (local address 127.0.98.62:43949)
0619 20:56:39.835502 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:39.835504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:39.836210 (+ 706us) server_negotiation.cc:407] Connection header received
0619 20:56:39.836987 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:39.836988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:39.837000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:39.837029 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:56:39.838157 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.838918 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.840413 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.840759 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.840804 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:39.843711 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:56:39.843712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:56:39.843713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:56:39.843721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:56:39.845528 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:39.846005 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:39.846007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:39.846007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:39.846079 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:39.846410 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:39.846411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:39.846411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:39.846548 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:56:39.846593 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:39.846983 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 20:56:39.847081 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:56:39.847889 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:39.834961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43949 (local address 127.0.98.2:44887)
0619 20:56:39.836081 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:39.836133 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:39.836869 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:39.837239 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:39.837243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:39.837727 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:56:39.838025 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:39.838032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.839035 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.839037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:39.840310 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:39.840312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:39.840896 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:39.840897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:39.841041 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:39.841661 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:56:39.841820 (+ 159us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:56:39.843589 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:56:39.845627 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:39.845633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:39.845636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:39.845914 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:39.846156 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:39.846157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:39.846157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:39.846326 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:39.846642 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:56:39.846647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:56:39.846827 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 20:56:39.847509 (+ 682us) client_negotiation.cc:240] Negotiation successful
0619 20:56:39.847668 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":95,"threads_started":1}
I20260619 20:56:39.848559 11955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43949
I20260619 20:56:39.848650 11955 heartbeater.cc:464] Registering TS with master...
I20260619 20:56:39.848827 11955 heartbeater.cc:511] Master 127.0.98.62:43949 requested a full tablet report, sending...
I20260619 20:56:39.849261 11605 ts_manager.cc:194] Registered new tserver with Master: c248719e3eb74abd8b27249147481b50 (127.0.98.2:43787)
I20260619 20:56:39.849726 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44887
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:56:39.875380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:56:39.993980 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:56:39.994329 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:56:39.994392 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:56:39.996107 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:56:39.996208 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:56:39.996274 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:56:39.998421 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:56:39.998816 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:56:39.999094 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:56:40.005506 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:40.006078 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:56:40.006309 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:56:40.007547 11960 server_base.cc:1034] running on GCE node
I20260619 20:56:40.007807 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:56:40.008126 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:56:40.009353 11960 hybrid_clock.cc:648] HybridClock initialized: now 1781902600009333 us; error 34 us; skew 500 ppm
Jun 19 20:56:40 dist-test-slave-8bgz krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:56:40.012810 11960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:56:40.013444 11960 webserver.cc:460] Webserver started at http://127.0.98.3:33995/ using document root <none> and password file <none>
I20260619 20:56:40.013761 11960 fs_manager.cc:359] Metadata directory not provided
I20260619 20:56:40.013886 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:56:40.014071 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:56:40 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:40.038378 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "9890891a8f0a4ad3a29a7479e6086842"
format_stamp: "Formatted at 2026-06-19 20:56:40 on dist-test-slave-8bgz"
server_key: "5dab441ea1904d75be7a9d8ae9510a0c"
server_key_iv: "dcd976faab6b44648042add49697d54a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:40.038906 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9890891a8f0a4ad3a29a7479e6086842"
format_stamp: "Formatted at 2026-06-19 20:56:40 on dist-test-slave-8bgz"
server_key: "5dab441ea1904d75be7a9d8ae9510a0c"
server_key_iv: "dcd976faab6b44648042add49697d54a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:40.046701 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260619 20:56:40.059578 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:40.060278 11960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:56:40.060405 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "9890891a8f0a4ad3a29a7479e6086842"
format_stamp: "Formatted at 2026-06-19 20:56:40 on dist-test-slave-8bgz"
server_key: "5dab441ea1904d75be7a9d8ae9510a0c"
server_key_iv: "dcd976faab6b44648042add49697d54a"
server_key_version: "kuduclusterkey@0"
I20260619 20:56:40.060549 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:56:40.088357 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:56:40.090575 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:56:40.090735 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:56:40.090997 11960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:56:40.091456 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:56:40.091549 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:40.091624 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:56:40.091681 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:56:40.098963 11960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35415
I20260619 20:56:40.099476 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:56:40.101235 12088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35415 every 8 connection(s)
I20260619 20:56:40.103380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 11960
I20260619 20:56:40.103482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:56:40.103669 392 external_mini_cluster.cc:1428] Setting key 77816e348bba675f9450b7a0c37b2026
Jun 19 20:56:40 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:56:40.111475 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.102041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58447 (local address 127.0.98.62:43949)
0619 20:56:40.102338 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.102340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.102406 (+ 66us) server_negotiation.cc:407] Connection header received
0619 20:56:40.102955 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.102956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.102968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.103025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:56:40.103723 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.104480 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.105849 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.106067 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.106152 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.108374 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:56:40.108375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:56:40.108375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:56:40.108383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:56:40.109965 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:40.110432 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:40.110433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:40.110433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:40.110488 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:40.110772 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:40.110773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:40.110774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:40.110885 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:56:40.110955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.111298 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.111381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260619 20:56:40.111938 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.102037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43949 (local address 127.0.98.3:58447)
0619 20:56:40.102253 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.102321 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.102840 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.103098 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.103102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.103352 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:56:40.103634 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.103639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.104579 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.104580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.105766 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.105769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.106162 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.106163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.106344 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.106891 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:56:40.106900 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:56:40.108296 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:56:40.110047 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:40.110065 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:40.110067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:40.110348 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:40.110584 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:56:40.110585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:56:40.110585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:56:40.110699 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:56:40.110957 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:56:40.110963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:56:40.111116 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.111564 (+ 448us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.111728 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260619 20:56:40.113437 12089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43949
I20260619 20:56:40.113544 12089 heartbeater.cc:464] Registering TS with master...
I20260619 20:56:40.113735 12089 heartbeater.cc:511] Master 127.0.98.62:43949 requested a full tablet report, sending...
I20260619 20:56:40.114142 11605 ts_manager.cc:194] Registered new tserver with Master: 9890891a8f0a4ad3a29a7479e6086842 (127.0.98.3:35415)
I20260619 20:56:40.114552 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58447
I20260619 20:56:40.116145 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:56:40.125067 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.117459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.0.98.62:43949)
0619 20:56:40.117495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.117497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.118267 (+ 770us) server_negotiation.cc:407] Connection header received
0619 20:56:40.118369 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.118370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.118380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.118411 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:56:40.119017 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.119675 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.120455 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.120804 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.120861 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.122316 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:56:40.122317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:56:40.122318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:56:40.122325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:56:40.123821 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:40.124198 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:40.124200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:40.124200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:40.124253 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:56:40.124459 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:56:40.124460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:56:40.124460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:56:40.124614 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:56:40.124690 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.124901 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.124974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:56:40.127291 11605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:56:40.129002 11605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:56:40.137254 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.0.1:48814)
0619 20:56:40.132121 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.132163 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.132229 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.133079 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.133080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.133082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:56:40.133124 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.133125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.134645 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.134646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.135701 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.135702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.136938 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.136939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.137027 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.137039 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.137117 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.137168 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260619 20:56:40.137758 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.98.1:43319)
0619 20:56:40.132354 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.132359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.132888 (+ 529us) server_negotiation.cc:407] Connection header received
0619 20:56:40.132926 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.132928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.132944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.133009 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:56:40.133193 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.134566 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.135780 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.136799 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.136853 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.136906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.137645 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.137689 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":45,"threads_started":1}
I20260619 20:56:40.138501 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43787 (local address 127.0.0.1:36340)
0619 20:56:40.132410 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.132418 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.132462 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.132750 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.132753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.132755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:56:40.132823 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.132825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.134316 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.134317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.135658 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.135660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.138262 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.138263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.138334 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.138338 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.138392 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.138430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260619 20:56:40.138594 11749 tablet_service.cc:1467] Processing CreateTablet for tablet 33755da0c74448f48fdd3827c204f85c (DEFAULT_TABLE table=test-table [id=226c5555f33f4a088917842aa0f163ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:56:40.138836 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.98.2:43787)
0619 20:56:40.132364 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.132368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.132515 (+ 147us) server_negotiation.cc:407] Connection header received
0619 20:56:40.132573 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.132575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.132592 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.132661 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:56:40.133007 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.134184 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.135779 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.136511 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.136602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.136692 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.138716 (+ 2024us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.138763 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":69,"threads_started":1}
I20260619 20:56:40.138953 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33755da0c74448f48fdd3827c204f85c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:56:40.141702 12107 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: Bootstrap starting.
I20260619 20:56:40.142470 12107 tablet_bootstrap.cc:654] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: Neither blocks nor log segments found. Creating new log.
I20260619 20:56:40.142951 11889 tablet_service.cc:1467] Processing CreateTablet for tablet 33755da0c74448f48fdd3827c204f85c (DEFAULT_TABLE table=test-table [id=226c5555f33f4a088917842aa0f163ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:56:40.143095 12107 log.cc:824] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: Log is configured to *not* fsync() on all Append() calls
I20260619 20:56:40.143265 11889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33755da0c74448f48fdd3827c204f85c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:56:40.146365 12109 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Bootstrap starting.
I20260619 20:56:40.147413 12109 tablet_bootstrap.cc:654] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Neither blocks nor log segments found. Creating new log.
I20260619 20:56:40.148164 12109 log.cc:824] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Log is configured to *not* fsync() on all Append() calls
I20260619 20:56:40.148821 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35415 (local address 127.0.0.1:38964)
0619 20:56:40.132120 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.132163 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.132229 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.132890 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.132891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.132892 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:56:40.132934 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.132936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.137875 (+ 4939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.137876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.146759 (+ 8883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.146762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.148566 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.148567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.148647 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.148652 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.148706 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.148745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":99,"threads_started":1}
I20260619 20:56:40.149101 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.131941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.0.98.3:35415)
0619 20:56:40.132706 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.132710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.132719 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:56:40.132752 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.132754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.132767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.132833 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:56:40.136731 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.137781 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.147197 (+ 9416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.148385 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.148457 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.148529 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.148974 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.149015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":74,"threads_started":1}
I20260619 20:56:40.149113 12107 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: No bootstrap required, opened a new log
I20260619 20:56:40.149336 12107 ts_tablet_manager.cc:1397] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 20:56:40.149679 12004 tablet_service.cc:1467] Processing CreateTablet for tablet 33755da0c74448f48fdd3827c204f85c (DEFAULT_TABLE table=test-table [id=226c5555f33f4a088917842aa0f163ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:56:40.150017 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33755da0c74448f48fdd3827c204f85c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:56:40.152050 12107 raft_consensus.cc:348] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.152820 12109 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: No bootstrap required, opened a new log
I20260619 20:56:40.152930 12109 ts_tablet_manager.cc:1397] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 20:56:40.153793 12107 raft_consensus.cc:374] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:56:40.153862 12107 raft_consensus.cc:729] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2a7171e0be4a9b8bb5e6f31049f887, State: Initialized, Role: FOLLOWER
I20260619 20:56:40.153990 12107 consensus_queue.cc:260] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.154296 12107 ts_tablet_manager.cc:1428] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 20:56:40.154431 11817 heartbeater.cc:503] Master 127.0.98.62:43949 was elected leader, sending a full tablet report...
I20260619 20:56:40.154502 12111 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Bootstrap starting.
I20260619 20:56:40.155460 12111 tablet_bootstrap.cc:654] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Neither blocks nor log segments found. Creating new log.
I20260619 20:56:40.155820 12111 log.cc:824] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Log is configured to *not* fsync() on all Append() calls
I20260619 20:56:40.156623 12111 tablet_bootstrap.cc:492] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: No bootstrap required, opened a new log
I20260619 20:56:40.156672 12111 ts_tablet_manager.cc:1397] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:56:40.157218 12109 raft_consensus.cc:348] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.157370 12109 raft_consensus.cc:374] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:56:40.157399 12109 raft_consensus.cc:729] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248719e3eb74abd8b27249147481b50, State: Initialized, Role: FOLLOWER
I20260619 20:56:40.157532 12109 consensus_queue.cc:260] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.157804 12109 ts_tablet_manager.cc:1428] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 20:56:40.157909 11955 heartbeater.cc:503] Master 127.0.98.62:43949 was elected leader, sending a full tablet report...
I20260619 20:56:40.158792 12111 raft_consensus.cc:348] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.158942 12111 raft_consensus.cc:374] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:56:40.158980 12111 raft_consensus.cc:729] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9890891a8f0a4ad3a29a7479e6086842, State: Initialized, Role: FOLLOWER
I20260619 20:56:40.159112 12111 consensus_queue.cc:260] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.159350 12111 ts_tablet_manager.cc:1428] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:56:40.159356 12089 heartbeater.cc:503] Master 127.0.98.62:43949 was elected leader, sending a full tablet report...
Jun 19 20:56:40 dist-test-slave-8bgz krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:56:40.254426 12114 raft_consensus.cc:482] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:56:40.254594 12114 raft_consensus.cc:504] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.255055 12114 leader_election.cc:288] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9890891a8f0a4ad3a29a7479e6086842 (127.0.98.3:35415), bb2a7171e0be4a9b8bb5e6f31049f887 (127.0.98.1:43319)
I20260619 20:56:40.259203 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.255179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35415 (local address 127.0.98.2:41993)
0619 20:56:40.255429 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.255441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.255485 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.255929 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.255930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.255932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:56:40.256000 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.256002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.256999 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.257000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.258180 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.258181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.258945 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.258946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.259011 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.259040 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.259094 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.259131 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":23}
I20260619 20:56:40.259630 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.255569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41993 (local address 127.0.98.3:35415)
0619 20:56:40.255674 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.255676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.255684 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:56:40.255710 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.255711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.255735 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.255786 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:56:40.256124 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.256881 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.258321 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.258843 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.258908 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.258920 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.259524 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.259564 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 20:56:40.259878 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33755da0c74448f48fdd3827c204f85c" candidate_uuid: "c248719e3eb74abd8b27249147481b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9890891a8f0a4ad3a29a7479e6086842" is_pre_election: true
I20260619 20:56:40.260010 12043 raft_consensus.cc:2454] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248719e3eb74abd8b27249147481b50 in term 0.
I20260619 20:56:40.260242 11842 leader_election.cc:302] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9890891a8f0a4ad3a29a7479e6086842, c248719e3eb74abd8b27249147481b50; no voters:
I20260619 20:56:40.260362 12114 raft_consensus.cc:2781] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:56:40.260394 12114 raft_consensus.cc:482] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:56:40.260414 12114 raft_consensus.cc:3037] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:56:40.261088 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.255631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42213 (local address 127.0.98.1:43319)
0619 20:56:40.255844 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.255846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.257437 (+ 1591us) server_negotiation.cc:407] Connection header received
0619 20:56:40.257680 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.257681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.257693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.257719 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:56:40.257900 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.258640 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.260063 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.260662 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.260711 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.260722 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.260964 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.261006 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260619 20:56:40.261344 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.255183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.98.2:42213)
0619 20:56:40.257379 (+ 2196us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.257390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:56:40.257460 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:56:40.257773 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:56:40.257774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:56:40.257776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:56:40.257834 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.257835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.258870 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.258871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.259962 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:56:40.259965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.260816 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.260817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:56:40.260885 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.260889 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:56:40.261196 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 20:56:40.261244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2177,"mutex_wait_us":13,"thread_start_us":92,"threads_started":1}
I20260619 20:56:40.261365 12114 raft_consensus.cc:504] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.261495 12114 leader_election.cc:288] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [CANDIDATE]: Term 1 election: Requested vote from peers 9890891a8f0a4ad3a29a7479e6086842 (127.0.98.3:35415), bb2a7171e0be4a9b8bb5e6f31049f887 (127.0.98.1:43319)
I20260619 20:56:40.261636 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33755da0c74448f48fdd3827c204f85c" candidate_uuid: "c248719e3eb74abd8b27249147481b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" is_pre_election: true
I20260619 20:56:40.261783 11761 raft_consensus.cc:2454] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248719e3eb74abd8b27249147481b50 in term 0.
I20260619 20:56:40.261988 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33755da0c74448f48fdd3827c204f85c" candidate_uuid: "c248719e3eb74abd8b27249147481b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9890891a8f0a4ad3a29a7479e6086842"
I20260619 20:56:40.262053 12043 raft_consensus.cc:3037] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:56:40.262938 12043 raft_consensus.cc:2454] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248719e3eb74abd8b27249147481b50 in term 1.
I20260619 20:56:40.263129 11842 leader_election.cc:302] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9890891a8f0a4ad3a29a7479e6086842, c248719e3eb74abd8b27249147481b50; no voters:
I20260619 20:56:40.263245 12114 raft_consensus.cc:2781] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:56:40.263403 12114 raft_consensus.cc:686] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 1 LEADER]: Becoming Leader. State: Replica: c248719e3eb74abd8b27249147481b50, State: Running, Role: LEADER
I20260619 20:56:40.263487 12114 consensus_queue.cc:237] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } }
I20260619 20:56:40.264308 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33755da0c74448f48fdd3827c204f85c" candidate_uuid: "c248719e3eb74abd8b27249147481b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887"
I20260619 20:56:40.264394 11761 raft_consensus.cc:3037] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:56:40.264509 11605 catalog_manager.cc:5495] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to c248719e3eb74abd8b27249147481b50 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c248719e3eb74abd8b27249147481b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c248719e3eb74abd8b27249147481b50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43787 } health_report { overall_health: HEALTHY } } }
I20260619 20:56:40.265192 11761 raft_consensus.cc:2454] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248719e3eb74abd8b27249147481b50 in term 1.
I20260619 20:56:40.289888 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.285856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.98.2:43787)
0619 20:56:40.285898 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:56:40.285900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:56:40.285909 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:56:40.285988 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:56:40.285989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:56:40.286012 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:56:40.286086 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:56:40.286305 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.287060 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.288703 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:56:40.288995 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:56:40.289054 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:56:40.289237 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:56:40.289367 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:56:40.289436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:56:40.289793 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 20:56:40.289812 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:56:40.292276 11761 raft_consensus.cc:1261] T 33755da0c74448f48fdd3827c204f85c P bb2a7171e0be4a9b8bb5e6f31049f887 [term 1 FOLLOWER]: Refusing update from remote peer c248719e3eb74abd8b27249147481b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:56:40.292608 12114 consensus_queue.cc:1035] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2a7171e0be4a9b8bb5e6f31049f887" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:56:40.294691 12043 raft_consensus.cc:1261] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 1 FOLLOWER]: Refusing update from remote peer c248719e3eb74abd8b27249147481b50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:56:40.294958 12114 consensus_queue.cc:1035] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9890891a8f0a4ad3a29a7479e6086842" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:56:40.296092 12120 mvcc.cc:205] Tried to move back new op lower bound from 7298673050794631168 to 7298673050679783424. Current Snapshot: MvccSnapshot[applied={T|T < 7298673050794631168}]
I20260619 20:56:40.296579 12121 mvcc.cc:205] Tried to move back new op lower bound from 7298673050794631168 to 7298673050679783424. Current Snapshot: MvccSnapshot[applied={T|T < 7298673050794631168}]
I20260619 20:56:40.297019 12125 mvcc.cc:205] Tried to move back new op lower bound from 7298673050794631168 to 7298673050679783424. Current Snapshot: MvccSnapshot[applied={T|T < 7298673050794631168}]
I20260619 20:56:40.300523 11605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:56:40.300640 11605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:56:40.301266 11605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aec9f2df610e45f29ff9e4424a055e70: Sending DeleteTablet for 3 replicas of tablet 33755da0c74448f48fdd3827c204f85c
I20260619 20:56:40.301626 11749 tablet_service.cc:1514] Processing DeleteTablet for tablet 33755da0c74448f48fdd3827c204f85c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48814
I20260619 20:56:40.301853 11889 tablet_service.cc:1514] Processing DeleteTablet for tablet 33755da0c74448f48fdd3827c204f85c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36340
I20260619 20:56:40.302438 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet 33755da0c74448f48fdd3827c204f85c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38964
I20260619 20:56:40.302659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 11686
I20260619 20:56:40.305902 12132 tablet_replica.cc:330] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: stopping tablet replica
I20260619 20:56:40.306010 12132 raft_consensus.cc:2229] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:56:40.306142 12132 raft_consensus.cc:2258] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:56:40.306607 12132 ts_tablet_manager.cc:1905] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:56:40.308218 12132 ts_tablet_manager.cc:1918] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:56:40.308287 12132 log.cc:1196] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/33755da0c74448f48fdd3827c204f85c
I20260619 20:56:40.308525 12132 ts_tablet_manager.cc:1939] T 33755da0c74448f48fdd3827c204f85c P 9890891a8f0a4ad3a29a7479e6086842: Deleting consensus metadata
I20260619 20:56:40.308960 11589 catalog_manager.cc:4841] TS 9890891a8f0a4ad3a29a7479e6086842 (127.0.98.3:35415): tablet 33755da0c74448f48fdd3827c204f85c (table test-table [id=226c5555f33f4a088917842aa0f163ca]) successfully deleted
I20260619 20:56:40.310319 12131 tablet_replica.cc:330] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: stopping tablet replica
I20260619 20:56:40.310412 12131 raft_consensus.cc:2229] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:56:40.310524 12131 raft_consensus.cc:2258] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:56:40.310927 12131 ts_tablet_manager.cc:1905] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:56:40.312343 12131 ts_tablet_manager.cc:1918] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:56:40.312407 12131 log.cc:1196] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/33755da0c74448f48fdd3827c204f85c
I20260619 20:56:40.312633 12131 ts_tablet_manager.cc:1939] T 33755da0c74448f48fdd3827c204f85c P c248719e3eb74abd8b27249147481b50: Deleting consensus metadata
I20260619 20:56:40.312935 11591 catalog_manager.cc:4841] TS c248719e3eb74abd8b27249147481b50 (127.0.98.2:43787): tablet 33755da0c74448f48fdd3827c204f85c (table test-table [id=226c5555f33f4a088917842aa0f163ca]) successfully deleted
W20260619 20:56:40.313483 11592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43319 (error 108)
I20260619 20:56:40.314002 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:56:40.313908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43319 (local address 127.0.0.1:48820)
0619 20:56:40.313948 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:56:40.313974 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 20:56:40.314172 11592 catalog_manager.cc:4584] TS bb2a7171e0be4a9b8bb5e6f31049f887 (127.0.98.1:43319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33755da0c74448f48fdd3827c204f85c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43319: connect: Connection refused (error 111)
I20260619 20:56:40.318490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 11823
I20260619 20:56:40.324664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 11960
I20260619 20:56:40.328696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 11573
I20260619 20:56:40.333966 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:56:40.667972 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:56:40.668061 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:56:41.012643 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:56:41.012 UTC [11159] LOG: received smart shutdown request
2026-06-19 20:56:41.016 UTC [11159] LOG: background worker "logical replication launcher" (PID 11168) exited with exit code 1
2026-06-19 20:56:41.016 UTC [11163] LOG: shutting down
2026-06-19 20:56:41.169 UTC [11159] LOG: database system is shut down
2026-06-19T20:56:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44216 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:56:41 dist-test-slave-8bgz krb5kdc[12142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:56:41 dist-test-slave-8bgz krb5kdc[12142](info): set up 2 sockets
Jun 19 20:56:41 dist-test-slave-8bgz krb5kdc[12142](info): commencing operation
krb5kdc: starting...
W20260619 20:56:42.314110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:56:42 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:56:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 20:56:42.464176 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_bKm47/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/postgres -l logfile start
2026-06-19 20:56:44.750 UTC [12172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:56:44.750 UTC [12172] LOG: listening on IPv4 address "127.0.98.20", port 52895
2026-06-19 20:56:44.753 UTC [12172] LOG: listening on Unix socket "/tmp/.s.PGSQL.52895"
2026-06-19 20:56:44.787 UTC [12175] LOG: database system was shut down at 2026-06-19 20:56:43 UTC
2026-06-19 20:56:44.791 UTC [12172] LOG: database system is ready to accept connections
I20260619 20:56:45.738035 392 mini_postgres.cc:96] Postgres bound to 52895
2026-06-19 20:56:45.743 UTC [12183] FATAL: database "slave" does not exist
127.0.98.20:52895 - accepting connections
I20260619 20:56:45.744098 392 mini_ranger.cc:161] Starting Ranger...
I20260619 20:56:45.758283 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:56:46.185755 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:56:46.185855 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin
2026-06-19 20:56:46,464 [I] DB FLAVOR :POSTGRES
2026-06-19 20:56:46,465 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:56:46,465 [I] Checking connection..
2026-06-19 20:56:46,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:56:46,688 [I] Checking connection passed.
2026-06-19 20:56:46,688 [I] --------- Verifying version history table ---------
2026-06-19 20:56:46,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:46,944 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:56:46,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:47,171 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:56:47,171 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:56:47,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:56:47.389 UTC [12247] WARNING: there is no transaction in progress
2026-06-19 20:56:47,395 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:56:47,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:56:47,644 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:56:47,645 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:56:47,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:56:47,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:56:48,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8bgz.c.gcp-upstream.internal','N') ;"
2026-06-19 20:56:48,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:56:48,575 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:56:48,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:56:48,823 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:56:48,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:56:49,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:56:49,266 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:56:49,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:56:50.207 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.223 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.238 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.252 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.266 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.284 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.338 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.344 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.358 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.368 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.378 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.397 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.406 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.412 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.420 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:50.425 UTC [12355] WARNING: there is no transaction in progress
2026-06-19 20:56:51,024 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:56:51,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:56:51,273 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:56:51,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:56:51,521 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:56:51,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:56:51,751 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:56:51,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:56:52,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:56:52,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8bgz.c.gcp-upstream.internal';"
2026-06-19 20:56:52,473 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:56:52,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:56:52,710 [I] DB_PATCHES have already been applied
I20260619 20:56:52.715976 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_bKm47/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:56:52.716032 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 20:56:52.718957 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57475
Jun 19, 2026 8:56:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:56:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:56:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:56:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:56:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:56:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:56:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:56:53.843257 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.099s user 0.000s sys 0.008s
Jun 19, 2026 8:57:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:57:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 8:57:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:57:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:57:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:57:13.814018 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:57:13 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902633, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:57:13.909907 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:57:13.919097 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:57:14.703277 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:57:14.703361 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms
I20260619 20:57:14.703383 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52895
2026-06-19 20:57:14,860 [I] DB FLAVOR :POSTGRES
2026-06-19 20:57:14,861 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:57:14,861 [I] Checking connection
2026-06-19 20:57:14,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:57:15,098 [I] connection success
2026-06-19 20:57:15,098 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:57:15,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:57:15,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:57:15,317 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:57:15,318 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:57:15,318 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:57:15,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:57:15,566 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:57:16.030555 392 mini_ranger_kms.cc:310] Created kms service
I20260619 20:57:16.198894 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:57:16.239048 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:57:16 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902633, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:16.611079 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:57:16.611194 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_bKm47/build/release/bin/postgresql.jar:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_bKm47/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:57:16.611212 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 20:57:16.614816 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12535
I20260619 20:57:16.614892 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44579
Jun 19, 2026 8:57:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:57:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:57:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:57:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:57:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:57:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 20:57:18 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902638, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:18.850418 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.941s user 0.000s sys 0.140s
I20260619 20:57:18.850521 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:57:18.850567 392 mini_ranger_kms.cc:401] 127.0.98.20:44579/kms/v1/keys
Jun 19 20:57:19 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902638, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:57:19 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902638, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:57:21 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:57:21.640895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41749
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:21.720163 12586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:21.720351 12586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:21.720372 12586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:21.721498 12586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:21.721539 12586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:21.721552 12586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:21.721604 12586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:21.721629 12586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:21.723290 12586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44579/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41749
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:21.723518 12586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:21.723743 12586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:21.729022 12592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:21.729036 12594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:21.729038 12591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:21.729086 12586 server_base.cc:1034] running on GCE node
I20260619 20:57:21.729384 12586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:21.729599 12586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:21.730774 12586 hybrid_clock.cc:648] HybridClock initialized: now 1781902641730762 us; error 29 us; skew 500 ppm
Jun 19 20:57:21 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:21.733129 12586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:21.733517 12586 webserver.cc:460] Webserver started at http://127.0.98.62:35457/ using document root <none> and password file <none>
I20260619 20:57:21.733714 12586 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:21.733752 12586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:21.733891 12586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:57:21 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:21.787034 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "e0addc3aa4444a00961173a13956d8c5"
format_stamp: "Formatted at 2026-06-19 20:57:21 on dist-test-slave-8bgz"
server_key: "7bb4b88cca7c085c6581c9fa6fd592af"
server_key_iv: "7c64875a05b997268015dbb3d084f10c"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:21.787434 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0addc3aa4444a00961173a13956d8c5"
format_stamp: "Formatted at 2026-06-19 20:57:21 on dist-test-slave-8bgz"
server_key: "7bb4b88cca7c085c6581c9fa6fd592af"
server_key_iv: "7c64875a05b997268015dbb3d084f10c"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:21.788885 12586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:57:21.799989 12601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:21.800200 12586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:21.800297 12586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "e0addc3aa4444a00961173a13956d8c5"
format_stamp: "Formatted at 2026-06-19 20:57:21 on dist-test-slave-8bgz"
server_key: "7bb4b88cca7c085c6581c9fa6fd592af"
server_key_iv: "7c64875a05b997268015dbb3d084f10c"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:21.800406 12586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:21.816578 12586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:21.818653 12586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:21.818823 12586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:21.823629 12586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41749
I20260619 20:57:21.823658 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41749 every 8 connection(s)
I20260619 20:57:21.824005 12586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:21.824652 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:21.826280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 12586
I20260619 20:57:21.826387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:57:21.826596 392 external_mini_cluster.cc:1428] Setting key 519e92a6e05622764fabe3d045ffb885
I20260619 20:57:21.827317 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Bootstrap starting.
I20260619 20:57:21.827877 12654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:21.828198 12654 log.cc:824] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:21.829031 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: No bootstrap required, opened a new log
I20260619 20:57:21.831200 12654 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:21.831400 12654 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:21.831467 12654 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0addc3aa4444a00961173a13956d8c5, State: Initialized, Role: FOLLOWER
I20260619 20:57:21.831614 12654 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:21.831717 12654 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:21.831770 12654 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:21.831840 12654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:21.832501 12654 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:21.832614 12654 leader_election.cc:302] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0addc3aa4444a00961173a13956d8c5; no voters:
I20260619 20:57:21.832765 12654 leader_election.cc:288] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:57:21.832803 12659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:21.833050 12659 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 LEADER]: Becoming Leader. State: Replica: e0addc3aa4444a00961173a13956d8c5, State: Running, Role: LEADER
I20260619 20:57:21.833112 12654 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:21.833187 12659 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:21.833649 12659 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0addc3aa4444a00961173a13956d8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } } }
I20260619 20:57:21.833797 12659 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:21.833995 12659 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0addc3aa4444a00961173a13956d8c5. Latest consensus state: current_term: 1 leader_uuid: "e0addc3aa4444a00961173a13956d8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } } }
I20260619 20:57:21.834108 12659 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: This master's current role is: LEADER
Jun 19 20:57:21 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:21.834759 12654 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:57:21.838166 12668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:21.839255 12668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:21.843384 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:21.827116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.98.62:41749)
0619 20:57:21.827392 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:21.827402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:21.827438 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:57:21.827989 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:21.827999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:21.828061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:21.828214 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:21.828913 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:21.830254 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:21.831397 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:21.832024 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:21.832133 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:21.837154 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:21.837205 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:21.837206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:21.837317 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:21.840790 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:21.841295 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:21.841298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:21.841298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:21.841416 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:21.841674 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:21.841675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:21.841676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:21.842264 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:21.842886 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:21.842967 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:57:21.843124 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
I20260619 20:57:21.847752 12668 catalog_manager.cc:1347] Generated new cluster ID: 130c9d87ceb745009bd47a26174a0dff
I20260619 20:57:21.847808 12668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:57:21.857353 12668 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:57:21.858143 12668 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:21.867859 12668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Generated new TSK 0
I20260619 20:57:21.868134 12668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:57:22.591141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:22.690645 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:22.690912 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:22.690953 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:22.692095 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:22.692135 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:22.692162 12699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:22.693635 12699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:22.693905 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:22.694133 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:22.700084 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:22.700340 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:22.700497 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:22.701067 12699 server_base.cc:1034] running on GCE node
I20260619 20:57:22.701267 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:22.701529 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:22.702719 12699 hybrid_clock.cc:648] HybridClock initialized: now 1781902642702697 us; error 34 us; skew 500 ppm
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:22.705926 12699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:22.706456 12699 webserver.cc:460] Webserver started at http://127.0.98.1:40071/ using document root <none> and password file <none>
I20260619 20:57:22.706717 12699 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:22.706782 12699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:22.706911 12699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:22.721382 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c050c3d96b14a869fd481eebc93341a"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "5542000de228c92560dd055f25c6401c"
server_key_iv: "3cb99023ead71283a2823c22d0b00eb7"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.721908 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c050c3d96b14a869fd481eebc93341a"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "5542000de228c92560dd055f25c6401c"
server_key_iv: "3cb99023ead71283a2823c22d0b00eb7"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.724078 12699 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:22.738343 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:22.741231 12699 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 20:57:22.741344 12699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "4c050c3d96b14a869fd481eebc93341a"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "5542000de228c92560dd055f25c6401c"
server_key_iv: "3cb99023ead71283a2823c22d0b00eb7"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.741443 12699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:22.760905 12699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:22.763155 12699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:22.763321 12699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:22.763586 12699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:22.763964 12699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:22.764016 12699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:22.764081 12699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:22.764125 12699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:22.771274 12699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35371
I20260619 20:57:22.771766 12699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:22.774647 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35371 every 8 connection(s)
I20260619 20:57:22.778411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 12699
I20260619 20:57:22.778515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:57:22.778717 392 external_mini_cluster.cc:1428] Setting key 7f682a27c802e30f4af72f750fec6a36
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:22.787849 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:22.775663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.1:40727)
0619 20:57:22.777181 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:22.777229 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:22.777924 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:22.778391 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:22.778396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:22.778642 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:22.778955 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:22.778962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:22.779983 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:22.779984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:22.781311 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:22.781314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:22.781872 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:22.781873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:22.782011 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:22.782627 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:22.782638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:22.784232 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:22.786246 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:22.786253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:22.786255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:22.786537 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:22.786754 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:22.786755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:22.786755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:22.786887 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:22.787189 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:22.787195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:22.787355 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 20:57:22.787459 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 20:57:22.787606 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":91,"threads_started":1}
I20260619 20:57:22.788170 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:22.775696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40727 (local address 127.0.98.62:41749)
0619 20:57:22.778112 (+ 2416us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:22.778114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:22.778122 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:22.778147 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:22.778148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:22.778160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:22.778256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:22.779060 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:22.779878 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:22.781406 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:22.781720 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:22.781793 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:22.784337 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:22.784339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:22.784339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:22.784347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:22.786134 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:22.786618 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:22.786619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:22.786619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:22.786680 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:22.786963 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:22.786964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:22.786964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:22.787104 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:22.787164 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:22.788001 (+ 837us) server_negotiation.cc:299] Negotiation successful
0619 20:57:22.788077 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2385,"thread_start_us":103,"threads_started":1}
I20260619 20:57:22.788630 12830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:22.788725 12830 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:22.788903 12830 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:22.789501 12618 ts_manager.cc:194] Registered new tserver with Master: 4c050c3d96b14a869fd481eebc93341a (127.0.98.1:35371)
I20260619 20:57:22.790398 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40727
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:57:22.817548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 20:57:22.937984 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:22.938264 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:22.938297 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:22.939903 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:22.939961 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:22.939998 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:22.942008 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:22.942286 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:22.942543 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:22.947866 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:22.947908 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:22.948247 12838 server_base.cc:1034] running on GCE node
W20260619 20:57:22.948335 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:22.948560 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:22.948851 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:22.952113 12838 hybrid_clock.cc:648] HybridClock initialized: now 1781902642952093 us; error 26 us; skew 500 ppm
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:22.954981 12838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:22.955549 12838 webserver.cc:460] Webserver started at http://127.0.98.2:33085/ using document root <none> and password file <none>
I20260619 20:57:22.955834 12838 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:22.955909 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:22.956040 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:57:22 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:22.969149 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff18f580feb74988af54f554204f67fb"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "018e7fb6444ff9256c8363b5a8304235"
server_key_iv: "94acaa234ed9ff127c10e8f2d9350758"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.969588 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff18f580feb74988af54f554204f67fb"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "018e7fb6444ff9256c8363b5a8304235"
server_key_iv: "94acaa234ed9ff127c10e8f2d9350758"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.971782 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:22.997372 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:22.997602 12838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260619 20:57:22.997685 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "ff18f580feb74988af54f554204f67fb"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "018e7fb6444ff9256c8363b5a8304235"
server_key_iv: "94acaa234ed9ff127c10e8f2d9350758"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:22.997784 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:23.010586 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:23.012974 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:23.013165 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:23.013391 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:23.013754 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:23.013827 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.013883 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:23.013926 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.020289 12838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36151
I20260619 20:57:23.020735 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:57:23.024260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 12838
I20260619 20:57:23.024349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:57:23.024547 392 external_mini_cluster.cc:1428] Setting key 2ba4559c6e65d30f46a9499f821a681f
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:23.034013 12967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36151 every 8 connection(s)
I20260619 20:57:23.037670 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.024632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36985 (local address 127.0.98.62:41749)
0619 20:57:23.024676 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:23.024678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:23.024929 (+ 251us) server_negotiation.cc:407] Connection header received
0619 20:57:23.025608 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:23.025609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:23.025621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:23.025651 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:23.026526 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.027293 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.028808 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.029144 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.029192 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.033928 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:23.033930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:23.033930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:23.033938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:23.035826 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.036316 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.036317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.036317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.036377 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.036867 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.036868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.036868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.037004 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:23.037052 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:23.037455 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 20:57:23.037550 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:57:23.038491 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.024330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.2:36985)
0619 20:57:23.024812 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:23.024856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:23.025489 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:23.025843 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:23.025848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:23.026090 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:23.026403 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.026410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.027415 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.027416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.028692 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.028695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.029316 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.029317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.029449 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.032174 (+ 2725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:23.032183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:23.033815 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:23.035928 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.035935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.035937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.036217 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.036454 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.036455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.036455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.036774 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.037102 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:23.037108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:23.037290 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 20:57:23.038073 (+ 783us) client_negotiation.cc:240] Negotiation successful
0619 20:57:23.038273 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":105,"threads_started":1}
I20260619 20:57:23.039165 12968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:23.039258 12968 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.039441 12968 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:23.039849 12618 ts_manager.cc:194] Registered new tserver with Master: ff18f580feb74988af54f554204f67fb (127.0.98.2:36151)
I20260619 20:57:23.040314 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:57:23.069927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:23.197656 12973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:23.198014 12973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:23.198092 12973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:23.199868 12973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:23.199965 12973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:23.200035 12973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:23.202576 12973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:23.202879 12973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:23.203140 12973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:23.209223 12981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.209501 12979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.209777 12978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.209980 12973 server_base.cc:1034] running on GCE node
I20260619 20:57:23.210259 12973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:23.210522 12973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:23.211709 12973 hybrid_clock.cc:648] HybridClock initialized: now 1781902643211687 us; error 30 us; skew 500 ppm
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:23.214959 12973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:23.215457 12973 webserver.cc:460] Webserver started at http://127.0.98.3:38465/ using document root <none> and password file <none>
I20260619 20:57:23.215694 12973 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:23.215749 12973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:23.215873 12973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:23.233386 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e49494fd4efb48969be28abb892e3e47"
format_stamp: "Formatted at 2026-06-19 20:57:23 on dist-test-slave-8bgz"
server_key: "bee7843591b9d0ce64484ca45e1eb861"
server_key_iv: "f31c768ff7769f579ae475964e8f2252"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.233835 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e49494fd4efb48969be28abb892e3e47"
format_stamp: "Formatted at 2026-06-19 20:57:23 on dist-test-slave-8bgz"
server_key: "bee7843591b9d0ce64484ca45e1eb861"
server_key_iv: "f31c768ff7769f579ae475964e8f2252"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.235935 12973 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 20:57:23.250353 12988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.250597 12973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 20:57:23.250702 12973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e49494fd4efb48969be28abb892e3e47"
format_stamp: "Formatted at 2026-06-19 20:57:23 on dist-test-slave-8bgz"
server_key: "bee7843591b9d0ce64484ca45e1eb861"
server_key_iv: "f31c768ff7769f579ae475964e8f2252"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.250802 12973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:23.268906 12973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:23.271281 12973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:23.271414 12973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:23.271617 12973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:23.271960 12973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:23.272002 12973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.272038 12973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:23.272055 12973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.278219 12973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33075
I20260619 20:57:23.278620 12973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:23.278759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 12973
I20260619 20:57:23.278860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:57:23.279060 392 external_mini_cluster.cc:1428] Setting key 94cdae1fbb93fae44e62668e7434924b
I20260619 20:57:23.287941 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33075 every 8 connection(s)
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:23.299111 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.288717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33279 (local address 127.0.98.62:41749)
0619 20:57:23.288773 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:23.288775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:23.289064 (+ 289us) server_negotiation.cc:407] Connection header received
0619 20:57:23.289682 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:23.289684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:23.289697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:23.289767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:23.290513 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.291324 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.292799 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.293094 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.293137 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.295831 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:23.295832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:23.295833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:23.295840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:23.297536 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.298003 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.298004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.298005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.298059 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.298379 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.298380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.298380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.298511 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:23.298555 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:23.298935 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 20:57:23.299017 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:57:23.299667 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.288716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.3:33279)
0619 20:57:23.288914 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:23.288964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:23.289572 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:23.289855 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:23.289860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:23.290103 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:23.290420 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.290426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.291439 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.291440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.292695 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.292698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.293275 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.293276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.293408 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.294034 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:23.294053 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:23.295719 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:23.297629 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.297635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.297638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.297919 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.298128 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.298129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.298129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.298295 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.298604 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:23.298609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:23.298780 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 20:57:23.299269 (+ 489us) client_negotiation.cc:240] Negotiation successful
0619 20:57:23.299435 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260619 20:57:23.300117 13102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:23.300228 13102 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.300410 13102 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:23.300866 12618 ts_manager.cc:194] Registered new tserver with Master: e49494fd4efb48969be28abb892e3e47 (127.0.98.3:33075)
I20260619 20:57:23.301404 12618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33279
I20260619 20:57:23.302361 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:57:23.303064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 12699
I20260619 20:57:23.309887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 12838
I20260619 20:57:23.315718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 12973
I20260619 20:57:23.321851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 12586
I20260619 20:57:23.330391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--webserver_interface=127.0.98.62
--webserver_port=35457
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41749
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:23.414989 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:23.415184 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:23.415205 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:23.416267 13105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:23.416301 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:23.416316 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:23.416328 13105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:23.416339 13105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:23.417804 13105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41749
--ranger_config_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:23.417982 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:23.418133 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:23.423091 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.423110 13110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.423193 13105 server_base.cc:1034] running on GCE node
W20260619 20:57:23.423092 13111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.423552 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:23.423894 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:23.425063 13105 hybrid_clock.cc:648] HybridClock initialized: now 1781902643425047 us; error 43 us; skew 500 ppm
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:23.427564 13105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:23.427983 13105 webserver.cc:460] Webserver started at http://127.0.98.62:35457/ using document root <none> and password file <none>
I20260619 20:57:23.428224 13105 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:23.428301 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:23.470165 13105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:23.471069 13121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.471292 13105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:23.471396 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "e0addc3aa4444a00961173a13956d8c5"
format_stamp: "Formatted at 2026-06-19 20:57:21 on dist-test-slave-8bgz"
server_key: "7bb4b88cca7c085c6581c9fa6fd592af"
server_key_iv: "7c64875a05b997268015dbb3d084f10c"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.471868 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:23.490132 13105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:23.492344 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:23.492491 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:23.497328 13105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41749
I20260619 20:57:23.497357 13173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41749 every 8 connection(s)
I20260619 20:57:23.497794 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:23.499027 13174 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:57:23.499629 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Bootstrap starting.
I20260619 20:57:23.506507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13105
I20260619 20:57:23.506834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35371
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41749
--webserver_port=40071
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:23.507534 13174 log.cc:824] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:23.509145 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:23.509490 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Bootstrap complete.
I20260619 20:57:23.511922 13174 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:23.512143 13174 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0addc3aa4444a00961173a13956d8c5, State: Initialized, Role: FOLLOWER
I20260619 20:57:23.512311 13174 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:23.512382 13174 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:23.512408 13174 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:23.512475 13174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:57:23.513451 13174 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:23.513533 13174 leader_election.cc:302] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0addc3aa4444a00961173a13956d8c5; no voters:
I20260619 20:57:23.513734 13174 leader_election.cc:288] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:57:23.513768 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:57:23.513870 13178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [term 2 LEADER]: Becoming Leader. State: Replica: e0addc3aa4444a00961173a13956d8c5, State: Running, Role: LEADER
I20260619 20:57:23.513952 13178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } }
I20260619 20:57:23.513986 13174 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:23.514247 13179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0addc3aa4444a00961173a13956d8c5. Latest consensus state: current_term: 2 leader_uuid: "e0addc3aa4444a00961173a13956d8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } } }
I20260619 20:57:23.514496 13179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:23.514423 13180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0addc3aa4444a00961173a13956d8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0addc3aa4444a00961173a13956d8c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41749 } } }
I20260619 20:57:23.514770 13184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:23.514779 13180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:23.515305 13184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:23.515451 13184 catalog_manager.cc:1259] Loaded cluster ID: 130c9d87ceb745009bd47a26174a0dff
I20260619 20:57:23.515493 13184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:57:23.515604 13174 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:57:23.519485 13184 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:23.519974 13184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Loaded TSK: 0
I20260619 20:57:23.520254 13184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:57:23.596386 13177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:23.596647 13177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:23.596679 13177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:23.598399 13177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:23.598448 13177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:23.598486 13177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:23.600602 13177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40071
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:23.600878 13177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:23.601153 13177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:23.608911 13214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.609122 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.609378 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.609484 13177 server_base.cc:1034] running on GCE node
I20260619 20:57:23.609835 13177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:23.610126 13177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:23.611286 13177 hybrid_clock.cc:648] HybridClock initialized: now 1781902643611257 us; error 42 us; skew 500 ppm
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:23.614614 13177 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:23.615283 13177 webserver.cc:460] Webserver started at http://127.0.98.1:40071/ using document root <none> and password file <none>
I20260619 20:57:23.615566 13177 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:23.615702 13177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:23.639392 13177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:23.640196 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.640400 13177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:57:23.640481 13177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "4c050c3d96b14a869fd481eebc93341a"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "5542000de228c92560dd055f25c6401c"
server_key_iv: "3cb99023ead71283a2823c22d0b00eb7"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.640906 13177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:23.662380 13177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:23.664861 13177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:23.665063 13177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:23.665354 13177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:23.665740 13177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:23.665830 13177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.665930 13177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:23.666002 13177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.672405 13177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35371
I20260619 20:57:23.672883 13177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:23.675293 13335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35371 every 8 connection(s)
I20260619 20:57:23.676784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13177
I20260619 20:57:23.677054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36151
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41749
--webserver_port=33085
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:23.689337 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.676019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.1:35331)
0619 20:57:23.676248 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:23.676289 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:23.676922 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:23.677723 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:23.677728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:23.678174 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:23.678653 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.678660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.680579 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.680580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.681777 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.681781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.682724 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.682725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.682841 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.683415 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:23.683424 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:23.685019 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:23.687600 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.687605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.687607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.687861 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.688103 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.688103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.688104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.688218 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.688633 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:23.688637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:23.688783 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 20:57:23.688877 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:57:23.689030 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260619 20:57:23.690282 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.676109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35331 (local address 127.0.98.62:41749)
0619 20:57:23.676318 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:23.676326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:23.676360 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:57:23.677208 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:23.677215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:23.677271 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:23.677396 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:23.678814 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.680457 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.681879 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.682532 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.682630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.685147 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:23.685151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:23.685151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:23.685162 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:23.687513 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.687940 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.687943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.687943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.688036 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.688281 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.688282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.688282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.688536 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:23.688606 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:23.689967 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0619 20:57:23.690071 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260619 20:57:23.691411 13336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:23.691495 13336 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.691655 13336 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:23.692473 13336 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41749 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:57:23.692550 13336 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.692610 13336 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:23.693073 13336 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.693136 13336 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:23.693482 13336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:57:23.797689 13340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:23.797973 13340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:23.798015 13340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:23.799711 13340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:23.799772 13340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:23.799810 13340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:23.801945 13340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33085
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:23.802199 13340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:23.802495 13340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:23.809058 13348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.809062 13345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.809619 13346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.810134 13340 server_base.cc:1034] running on GCE node
I20260619 20:57:23.810411 13340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:23.810729 13340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:23.811937 13340 hybrid_clock.cc:648] HybridClock initialized: now 1781902643811924 us; error 45 us; skew 500 ppm
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:23.814731 13340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:23.815251 13340 webserver.cc:460] Webserver started at http://127.0.98.2:33085/ using document root <none> and password file <none>
I20260619 20:57:23.815523 13340 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:23.815605 13340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:23.835354 13340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:23.838588 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.838914 13340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 20:57:23.839032 13340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "ff18f580feb74988af54f554204f67fb"
format_stamp: "Formatted at 2026-06-19 20:57:22 on dist-test-slave-8bgz"
server_key: "018e7fb6444ff9256c8363b5a8304235"
server_key_iv: "94acaa234ed9ff127c10e8f2d9350758"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:23.839504 13340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:23.852344 13340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:23.854498 13340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:23.854661 13340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:23.854919 13340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:23.855290 13340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:23.855345 13340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.855382 13340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:23.855406 13340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:23.862479 13340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36151
I20260619 20:57:23.862596 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36151 every 8 connection(s)
I20260619 20:57:23.862911 13340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:57:23.865236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13340
I20260619 20:57:23.865522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33075
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41749
--webserver_port=38465
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:23.873297 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.863613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41927 (local address 127.0.98.62:41749)
0619 20:57:23.863659 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:23.863662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:23.863924 (+ 262us) server_negotiation.cc:407] Connection header received
0619 20:57:23.864447 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:23.864448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:23.864465 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:23.864521 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:23.865050 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.865721 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.867092 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.867392 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.867442 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.869998 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:23.869999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:23.870000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:23.870008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:23.871809 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.872310 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.872311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.872311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.872377 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:23.872605 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:23.872606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:23.872606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:23.872720 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:23.872812 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:23.873114 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 20:57:23.873208 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:57:23.873636 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:23.863624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.2:41927)
0619 20:57:23.863785 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:23.863823 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:23.864341 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:23.864592 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:23.864596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:23.864769 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:23.864957 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.864961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.865826 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.865827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.866980 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:23.866983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:23.867544 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:23.867545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:23.867700 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:23.868263 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:23.868274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:23.869887 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:23.871929 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.871935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.871937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.872214 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.872435 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:23.872435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:23.872436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:23.872532 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:23.872803 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:23.872809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:23.872978 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 20:57:23.873083 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:57:23.873223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260619 20:57:23.874122 13470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:23.874251 13470 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.874454 13470 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:23.875080 13470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41749 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:57:23.875155 13470 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.875222 13470 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:23.875551 13470 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:23.875633 13470 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:23.875941 13470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:57:23.986352 13473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:23.986608 13473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:23.986642 13473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:23.988241 13473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:23.988294 13473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:23.988332 13473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:23.990383 13473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38465
--tserver_master_addrs=127.0.98.62:41749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:23.990625 13473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:23.990883 13473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:23.997530 13478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:23.997709 13481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.998010 13473 server_base.cc:1034] running on GCE node
W20260619 20:57:23.998101 13479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:23.998373 13473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:23.998660 13473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:23.999861 13473 hybrid_clock.cc:648] HybridClock initialized: now 1781902643999830 us; error 43 us; skew 500 ppm
Jun 19 20:57:23 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:24.003132 13473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:24.003649 13473 webserver.cc:460] Webserver started at http://127.0.98.3:38465/ using document root <none> and password file <none>
I20260619 20:57:24.003907 13473 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:24.003986 13473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:57:24 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:24.025484 13473 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.001s
I20260619 20:57:24.026588 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:24.026757 13473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:57:24.026849 13473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e49494fd4efb48969be28abb892e3e47"
format_stamp: "Formatted at 2026-06-19 20:57:23 on dist-test-slave-8bgz"
server_key: "bee7843591b9d0ce64484ca45e1eb861"
server_key_iv: "f31c768ff7769f579ae475964e8f2252"
server_key_version: "kuduclusterkey@0"
I20260619 20:57:24.027334 13473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:24.044005 13473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:24.046582 13473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:24.046768 13473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:24.047024 13473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:24.047379 13473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:24.047430 13473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:24.047466 13473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:24.047497 13473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:24.059834 13473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33075
I20260619 20:57:24.060086 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33075 every 8 connection(s)
I20260619 20:57:24.060316 13473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:24.064528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13473
Jun 19 20:57:24 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:24.072871 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:24.061254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33659 (local address 127.0.98.62:41749)
0619 20:57:24.061316 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:24.061318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:24.061583 (+ 265us) server_negotiation.cc:407] Connection header received
0619 20:57:24.062398 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:24.062399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:24.062414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:24.062451 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:24.063290 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.063776 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.065508 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.065850 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.065925 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:24.068618 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:24.068620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:24.068620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:24.068628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:24.070486 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:24.071009 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:24.071010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:24.071010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:24.071201 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:24.071532 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:24.071533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:24.071533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:24.071678 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:24.071761 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:24.072653 (+ 892us) server_negotiation.cc:299] Negotiation successful
0619 20:57:24.072767 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:57:24.073110 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:24.061241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41749 (local address 127.0.98.3:33659)
0619 20:57:24.061423 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:24.061471 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:24.062250 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:24.062655 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:24.062660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:24.062886 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:24.063176 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:24.063183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.063881 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.063882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:24.065420 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:24.065423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.065978 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.065979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:24.066129 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:24.066794 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:24.066804 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:24.068512 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:24.070578 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:24.070585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:24.070587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:24.070910 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:24.071300 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:24.071301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:24.071301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:24.071451 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:24.071792 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:24.071798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:24.072224 (+ 426us) client_negotiation.cc:769] Sending connection context
0619 20:57:24.072719 (+ 495us) client_negotiation.cc:240] Negotiation successful
0619 20:57:24.072881 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260619 20:57:24.073557 13604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41749
I20260619 20:57:24.073766 13604 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:24.073982 13604 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:24.074860 13604 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41749 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:57:24.074990 13604 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:24.075127 13604 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:24.075481 13604 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:24.075591 13604 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
W20260619 20:57:24.075918 13604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:57:24.076866 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:24.064935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.0.98.62:41749)
0619 20:57:24.065047 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:24.065048 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:24.065199 (+ 151us) server_negotiation.cc:407] Connection header received
0619 20:57:24.065279 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:24.065281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:24.065292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:24.065383 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:24.066284 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.066965 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.068022 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:24.068340 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:24.068424 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:24.071572 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:24.071574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:24.071574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:24.071581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:24.073783 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:24.075720 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:24.075721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:24.075721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:24.075837 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:24.076318 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:24.076319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:24.076320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:24.076473 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:24.076634 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:24.076742 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:57:24.076808 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
Jun 19 20:57:24 dist-test-slave-8bgz krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:24.693666 13336 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:24.693820 13336 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:24.694268 13138 ts_manager.cc:194] Registered new tserver with Master: 4c050c3d96b14a869fd481eebc93341a (127.0.98.1:35371)
I20260619 20:57:24.695279 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35331
I20260619 20:57:24.876135 13470 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:24.876277 13470 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:24.876585 13138 ts_manager.cc:194] Registered new tserver with Master: ff18f580feb74988af54f554204f67fb (127.0.98.2:36151)
I20260619 20:57:24.877080 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41927
Jun 19 20:57:25 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:25.076143 13604 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:25.076303 13604 heartbeater.cc:511] Master 127.0.98.62:41749 requested a full tablet report, sending...
I20260619 20:57:25.076596 13138 ts_manager.cc:194] Registered new tserver with Master: e49494fd4efb48969be28abb892e3e47 (127.0.98.3:33075)
I20260619 20:57:25.077174 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33659
I20260619 20:57:25.080880 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:57:25.092267 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.081839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.98.62:41749)
0619 20:57:25.082276 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.082277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.082285 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:25.082307 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.082308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.082319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.083055 (+ 736us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:25.083107 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.083875 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.084976 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.085290 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.085621 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.089430 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:25.089432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:25.089432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:25.089439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:25.091108 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:25.091483 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:25.091484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:25.091484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:25.091549 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:25.091749 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:25.091750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:25.091751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:25.091882 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:25.091976 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.092084 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.092164 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":67,"threads_started":1}
I20260619 20:57:25.094301 13138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:57:25.096604 13138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:57:25.104692 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.099541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36151 (local address 127.0.0.1:54490)
0619 20:57:25.099924 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.099960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:25.100021 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:25.100857 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:25.100861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:25.100863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:25.100931 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.100932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.102190 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.102191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.103469 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.103471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104424 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.104425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.104498 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.104511 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:57:25.104566 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:57:25.104612 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":101,"threads_started":1}
I20260619 20:57:25.105314 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.100039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.98.2:36151)
0619 20:57:25.100666 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.100677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.100688 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:57:25.100724 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.100737 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.100754 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.100846 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:25.100995 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.102098 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.103567 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104338 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.105095 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.105176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.105201 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.105238 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":53,"threads_started":1}
I20260619 20:57:25.106127 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 (DEFAULT_TABLE table=test-table [id=c05f828ccb3146acb7a2dfc9f9b7f637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:25.106334 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.100201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.0.98.3:33075)
0619 20:57:25.100341 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.100346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.102753 (+ 2407us) server_negotiation.cc:407] Connection header received
0619 20:57:25.102789 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.102792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.102805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.102867 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:25.103035 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104014 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.105370 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.105971 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.106078 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.106128 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.106227 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.106262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260619 20:57:25.106516 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c5ea217a6b4bbd8f8846c4b1258392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:25.106539 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.099781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33075 (local address 127.0.0.1:41152)
0619 20:57:25.102650 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.102658 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:25.102698 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:25.102937 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:25.102938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:25.102939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:25.102973 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.102974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104113 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.104114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.105278 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.105281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.106073 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.106074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.106132 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.106138 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:57:25.106190 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:57:25.106263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2856,"mutex_wait_us":60,"thread_start_us":64,"threads_started":1}
I20260619 20:57:25.109881 13637 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Bootstrap starting.
I20260619 20:57:25.110603 13637 tablet_bootstrap.cc:654] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:25.110560 13538 tablet_service.cc:1467] Processing CreateTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 (DEFAULT_TABLE table=test-table [id=c05f828ccb3146acb7a2dfc9f9b7f637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:25.110895 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c5ea217a6b4bbd8f8846c4b1258392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:25.111104 13637 log.cc:824] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:25.111485 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.099542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35371 (local address 127.0.0.1:43654)
0619 20:57:25.102494 (+ 2952us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.102505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:25.102574 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:25.102829 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:25.102830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:25.102833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:25.102874 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.102875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104095 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.104096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.105660 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.105663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.111217 (+ 5554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.111218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.111294 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.111299 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:57:25.111353 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:57:25.111396 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2935,"mutex_wait_us":50,"thread_start_us":93,"threads_started":1}
I20260619 20:57:25.111793 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.099794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.1:35371)
0619 20:57:25.100132 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.100136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.102601 (+ 2465us) server_negotiation.cc:407] Connection header received
0619 20:57:25.102644 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.102646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.102662 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.102741 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:25.102947 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.104002 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.110289 (+ 6287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.111040 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.111129 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.111183 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.111657 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.111706 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":44,"threads_started":1}
I20260619 20:57:25.112069 13637 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: No bootstrap required, opened a new log
I20260619 20:57:25.112128 13637 ts_tablet_manager.cc:1397] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:57:25.112421 13270 tablet_service.cc:1467] Processing CreateTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 (DEFAULT_TABLE table=test-table [id=c05f828ccb3146acb7a2dfc9f9b7f637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:25.112756 13270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c5ea217a6b4bbd8f8846c4b1258392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:25.114648 13637 raft_consensus.cc:348] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.114799 13637 raft_consensus.cc:374] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:25.114831 13637 raft_consensus.cc:729] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff18f580feb74988af54f554204f67fb, State: Initialized, Role: FOLLOWER
I20260619 20:57:25.114876 13639 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Bootstrap starting.
I20260619 20:57:25.115208 13637 consensus_queue.cc:260] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.115538 13637 ts_tablet_manager.cc:1428] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:57:25.115602 13639 tablet_bootstrap.cc:654] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:25.115696 13470 heartbeater.cc:503] Master 127.0.98.62:41749 was elected leader, sending a full tablet report...
I20260619 20:57:25.116014 13639 log.cc:824] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:25.116343 13641 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Bootstrap starting.
I20260619 20:57:25.116773 13639 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: No bootstrap required, opened a new log
I20260619 20:57:25.116850 13639 ts_tablet_manager.cc:1397] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:25.117002 13641 tablet_bootstrap.cc:654] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:25.117283 13641 log.cc:824] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:25.118044 13641 tablet_bootstrap.cc:492] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: No bootstrap required, opened a new log
I20260619 20:57:25.118129 13641 ts_tablet_manager.cc:1397] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:25.119820 13639 raft_consensus.cc:348] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.120012 13639 raft_consensus.cc:374] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:25.119921 13641 raft_consensus.cc:348] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.120054 13639 raft_consensus.cc:729] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49494fd4efb48969be28abb892e3e47, State: Initialized, Role: FOLLOWER
I20260619 20:57:25.120076 13641 raft_consensus.cc:374] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:25.120113 13641 raft_consensus.cc:729] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c050c3d96b14a869fd481eebc93341a, State: Initialized, Role: FOLLOWER
I20260619 20:57:25.120191 13639 consensus_queue.cc:260] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.120230 13641 consensus_queue.cc:260] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.120460 13639 ts_tablet_manager.cc:1428] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 20:57:25.120484 13604 heartbeater.cc:503] Master 127.0.98.62:41749 was elected leader, sending a full tablet report...
I20260619 20:57:25.120496 13641 ts_tablet_manager.cc:1428] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:57:25.120625 13336 heartbeater.cc:503] Master 127.0.98.62:41749 was elected leader, sending a full tablet report...
W20260619 20:57:25.176753 13337 tablet.cc:2369] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:25.243042 13645 raft_consensus.cc:482] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:25.243224 13645 raft_consensus.cc:504] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.243640 13645 leader_election.cc:288] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff18f580feb74988af54f554204f67fb (127.0.98.2:36151), e49494fd4efb48969be28abb892e3e47 (127.0.98.3:33075)
I20260619 20:57:25.247776 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.244004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51475 (local address 127.0.98.2:36151)
0619 20:57:25.244053 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.244055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.244068 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:57:25.244187 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.244189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.244201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.244232 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:25.244476 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.245260 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.246803 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.247364 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.247413 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.247424 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.247648 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.247690 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:57:25.248049 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.243829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36151 (local address 127.0.98.1:51475)
0619 20:57:25.244002 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.244014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:25.244092 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:25.244300 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:25.244302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:25.244304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:25.244395 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.244397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.245487 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.245488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.246697 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.246701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.247484 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.247485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.247551 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.247555 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:25.247910 (+ 355us) client_negotiation.cc:240] Negotiation successful
0619 20:57:25.247953 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260619 20:57:25.248440 13405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c5ea217a6b4bbd8f8846c4b1258392" candidate_uuid: "4c050c3d96b14a869fd481eebc93341a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff18f580feb74988af54f554204f67fb" is_pre_election: true
I20260619 20:57:25.248620 13405 raft_consensus.cc:2454] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c050c3d96b14a869fd481eebc93341a in term 0.
I20260619 20:57:25.248899 13224 leader_election.cc:302] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c050c3d96b14a869fd481eebc93341a, ff18f580feb74988af54f554204f67fb; no voters:
I20260619 20:57:25.249023 13645 raft_consensus.cc:2781] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:57:25.249063 13645 raft_consensus.cc:482] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:57:25.249084 13645 raft_consensus.cc:3037] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:25.250013 13645 raft_consensus.cc:504] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.250138 13645 leader_election.cc:288] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [CANDIDATE]: Term 1 election: Requested vote from peers ff18f580feb74988af54f554204f67fb (127.0.98.2:36151), e49494fd4efb48969be28abb892e3e47 (127.0.98.3:33075)
I20260619 20:57:25.250496 13405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c5ea217a6b4bbd8f8846c4b1258392" candidate_uuid: "4c050c3d96b14a869fd481eebc93341a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff18f580feb74988af54f554204f67fb"
I20260619 20:57:25.250573 13405 raft_consensus.cc:3037] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:25.251420 13405 raft_consensus.cc:2454] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c050c3d96b14a869fd481eebc93341a in term 1.
I20260619 20:57:25.251644 13224 leader_election.cc:302] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c050c3d96b14a869fd481eebc93341a, ff18f580feb74988af54f554204f67fb; no voters:
I20260619 20:57:25.251878 13645 raft_consensus.cc:2781] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:25.252012 13645 raft_consensus.cc:686] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [term 1 LEADER]: Becoming Leader. State: Replica: 4c050c3d96b14a869fd481eebc93341a, State: Running, Role: LEADER
I20260619 20:57:25.252094 13645 consensus_queue.cc:237] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } }
I20260619 20:57:25.252894 13137 catalog_manager.cc:5495] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c050c3d96b14a869fd481eebc93341a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4c050c3d96b14a869fd481eebc93341a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c050c3d96b14a869fd481eebc93341a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260619 20:57:25.257795 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.244217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33075 (local address 127.0.98.1:52657)
0619 20:57:25.248127 (+ 3910us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.248139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:25.248217 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:25.248396 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:25.248397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:25.248399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:25.248426 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.248427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.249318 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.249318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.250295 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:25.250299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.257572 (+ 7273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.257573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:25.257642 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.257646 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:25.257688 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 20:57:25.257728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":80,"threads_started":1}
I20260619 20:57:25.258370 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.244397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52657 (local address 127.0.98.3:33075)
0619 20:57:25.244443 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.244445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.248189 (+ 3744us) server_negotiation.cc:407] Connection header received
0619 20:57:25.248292 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.248293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.248306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.248335 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:25.248498 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.249201 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.250394 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.250878 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.250927 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.250936 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.258254 (+ 7318us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.258305 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:57:25.258644 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c5ea217a6b4bbd8f8846c4b1258392" candidate_uuid: "4c050c3d96b14a869fd481eebc93341a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49494fd4efb48969be28abb892e3e47" is_pre_election: true
I20260619 20:57:25.258782 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c5ea217a6b4bbd8f8846c4b1258392" candidate_uuid: "4c050c3d96b14a869fd481eebc93341a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49494fd4efb48969be28abb892e3e47"
I20260619 20:57:25.258795 13558 raft_consensus.cc:2454] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c050c3d96b14a869fd481eebc93341a in term 0.
Jun 19 20:57:25 dist-test-slave-8bgz krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:57:25.261155 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.257415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.98.1:35371)
0619 20:57:25.257453 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:25.257455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:25.257464 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:57:25.257534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:25.257534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:25.257551 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:25.257622 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:57:25.257808 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.259095 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.260213 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:25.260480 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:25.260550 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:25.260790 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:57:25.260897 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:57:25.260959 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:25.261056 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:57:25.261083 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:57:25.262974 13558 raft_consensus.cc:3037] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:25.263060 13405 raft_consensus.cc:1261] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 1 FOLLOWER]: Refusing update from remote peer 4c050c3d96b14a869fd481eebc93341a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:25.263407 13649 consensus_queue.cc:1035] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff18f580feb74988af54f554204f67fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:25.263859 13558 raft_consensus.cc:1261] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 1 FOLLOWER]: Refusing update from remote peer 4c050c3d96b14a869fd481eebc93341a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:25.264130 13649 consensus_queue.cc:1035] T 57c5ea217a6b4bbd8f8846c4b1258392 P 4c050c3d96b14a869fd481eebc93341a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49494fd4efb48969be28abb892e3e47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:25.268168 13653 mvcc.cc:205] Tried to move back new op lower bound from 7298673234994831360 to 7298673234953166848. Current Snapshot: MvccSnapshot[applied={T|T < 7298673234994831360}]
I20260619 20:57:25.269389 13652 mvcc.cc:205] Tried to move back new op lower bound from 7298673234994831360 to 7298673234953166848. Current Snapshot: MvccSnapshot[applied={T|T < 7298673234994831360}]
I20260619 20:57:25.272166 13137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:25.272257 13137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:25.272921 13137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0addc3aa4444a00961173a13956d8c5: Sending DeleteTablet for 3 replicas of tablet 57c5ea217a6b4bbd8f8846c4b1258392
I20260619 20:57:25.273222 13538 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41152
I20260619 20:57:25.273238 13270 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43654
I20260619 20:57:25.273440 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c5ea217a6b4bbd8f8846c4b1258392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54490
I20260619 20:57:25.273658 13663 tablet_replica.cc:330] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: stopping tablet replica
I20260619 20:57:25.273803 13663 raft_consensus.cc:2229] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:25.273914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13177
I20260619 20:57:25.273950 13663 raft_consensus.cc:2258] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:25.275512 13663 ts_tablet_manager.cc:1905] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:25.280411 13661 tablet_replica.cc:330] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: stopping tablet replica
I20260619 20:57:25.280526 13661 raft_consensus.cc:2229] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:25.280624 13661 raft_consensus.cc:2258] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:25.281126 13661 ts_tablet_manager.cc:1905] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:25.282697 13661 ts_tablet_manager.cc:1918] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:25.282771 13661 log.cc:1196] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/57c5ea217a6b4bbd8f8846c4b1258392
I20260619 20:57:25.283037 13661 ts_tablet_manager.cc:1939] T 57c5ea217a6b4bbd8f8846c4b1258392 P e49494fd4efb48969be28abb892e3e47: Deleting consensus metadata
I20260619 20:57:25.283452 13124 catalog_manager.cc:4841] TS e49494fd4efb48969be28abb892e3e47 (127.0.98.3:33075): tablet 57c5ea217a6b4bbd8f8846c4b1258392 (table test-table [id=c05f828ccb3146acb7a2dfc9f9b7f637]) successfully deleted
I20260619 20:57:25.286357 13663 ts_tablet_manager.cc:1918] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:25.286440 13663 log.cc:1196] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/57c5ea217a6b4bbd8f8846c4b1258392
I20260619 20:57:25.286737 13663 ts_tablet_manager.cc:1939] T 57c5ea217a6b4bbd8f8846c4b1258392 P ff18f580feb74988af54f554204f67fb: Deleting consensus metadata
I20260619 20:57:25.287175 13123 catalog_manager.cc:4841] TS ff18f580feb74988af54f554204f67fb (127.0.98.2:36151): tablet 57c5ea217a6b4bbd8f8846c4b1258392 (table test-table [id=c05f828ccb3146acb7a2dfc9f9b7f637]) successfully deleted
I20260619 20:57:25.287990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13340
W20260619 20:57:25.288844 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35371 (error 108)
I20260619 20:57:25.289345 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:25.289209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35371 (local address 127.0.0.1:43658)
0619 20:57:25.289269 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:25.289311 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:57:25.294416 13122 catalog_manager.cc:4584] TS 4c050c3d96b14a869fd481eebc93341a (127.0.98.1:35371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57c5ea217a6b4bbd8f8846c4b1258392: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35371: connect: Connection refused (error 111)
I20260619 20:57:25.295220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13473
I20260619 20:57:25.301595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13105
I20260619 20:57:25.308382 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:57:25.642508 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:57:25.642601 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:57:25.987793 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:57:25.987 UTC [12172] LOG: received smart shutdown request
2026-06-19 20:57:25.991 UTC [12172] LOG: background worker "logical replication launcher" (PID 12181) exited with exit code 1
2026-06-19 20:57:25.991 UTC [12176] LOG: shutting down
2026-06-19 20:57:26.139 UTC [12172] LOG: database system is shut down
2026-06-19T20:57:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44973 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:57:26 dist-test-slave-8bgz krb5kdc[13674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:57:26 dist-test-slave-8bgz krb5kdc[13674](info): set up 2 sockets
Jun 19 20:57:26 dist-test-slave-8bgz krb5kdc[13674](info): commencing operation
krb5kdc: starting...
W20260619 20:57:27.278059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:57:27.388477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:27.467532 13687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:27.467712 13687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:27.467733 13687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:27.468784 13687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:27.468820 13687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:27.468834 13687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:27.468847 13687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:27.468895 13687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:27.470372 13687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:27.470609 13687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:27.470839 13687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:27.475389 13693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.475394 13692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.475397 13695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:27.475543 13687 server_base.cc:1034] running on GCE node
I20260619 20:57:27.475839 13687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:27.476115 13687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:27.477277 13687 hybrid_clock.cc:648] HybridClock initialized: now 1781902647477255 us; error 34 us; skew 500 ppm
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:27.479609 13687 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:27.480082 13687 webserver.cc:460] Webserver started at http://127.0.98.62:43413/ using document root <none> and password file <none>
I20260619 20:57:27.480284 13687 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:27.480346 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:27.480499 13687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:27.481524 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "ee0066e2b8cb4525bf64c4ec3046cb10"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "00bb5a0243d336078d8b8d286514a609"
server_key_iv: "d58133c451927477a77e2fda53696d26"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.481884 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee0066e2b8cb4525bf64c4ec3046cb10"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "00bb5a0243d336078d8b8d286514a609"
server_key_iv: "d58133c451927477a77e2fda53696d26"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.483188 13687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:57:27.483860 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:27.484015 13687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:27.484120 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "ee0066e2b8cb4525bf64c4ec3046cb10"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "00bb5a0243d336078d8b8d286514a609"
server_key_iv: "d58133c451927477a77e2fda53696d26"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.484191 13687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:27.494063 13687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:27.495821 13687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:27.495976 13687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:27.500204 13687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46097
I20260619 20:57:27.500250 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46097 every 8 connection(s)
I20260619 20:57:27.500590 13687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:27.501358 13755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:27.502418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13687
I20260619 20:57:27.502534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:57:27.502704 392 external_mini_cluster.cc:1428] Setting key 2a91702869f91c2da7a1a7024f3e8c23
I20260619 20:57:27.502990 13755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 20:57:27.512624 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.503659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46097 (local address 127.0.0.1:42370)
0619 20:57:27.503824 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.503861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.504307 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.504610 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.504613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.504873 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.505024 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.505028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.506016 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.506016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.506754 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.506756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.507250 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.507250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.507313 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.507798 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.507805 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.509256 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.511132 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.511138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.511139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.511337 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.511622 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.511624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.511624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.511716 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.512086 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.512090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.512148 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.512242 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.512447 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":87,"thread_start_us":78,"threads_started":1}
I20260619 20:57:27.512624 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.503676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.0.98.62:46097)
0619 20:57:27.503844 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.503852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.503937 (+ 85us) server_negotiation.cc:407] Connection header received
0619 20:57:27.504473 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.504475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.504508 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.504604 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.505125 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.505907 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.506846 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.507155 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.507278 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.509369 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.509376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.509376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.509382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.511030 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.511433 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.511436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.511436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.511513 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.511805 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.511806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.511806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.512001 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.512106 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.512263 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.512392 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260619 20:57:27.513526 13755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:57:27.513806 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.513724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55784)
0619 20:57:27.513753 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.513789 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 20:57:27.513957 13703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
I20260619 20:57:27.514379 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.514309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55796)
0619 20:57:27.514341 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.514364 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 20:57:27.514498 13755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:57:27.525480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42817
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:27.563737 13755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } attempt: 1
I20260619 20:57:27.564266 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.564135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55808)
0619 20:57:27.564196 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.564239 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:57:27.564625 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.564536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55810)
0619 20:57:27.564586 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.564610 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:57:27.564723 13755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
W20260619 20:57:27.604207 13762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:27.604395 13762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:27.604417 13762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:27.605453 13762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:27.605486 13762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:27.605499 13762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:27.605512 13762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:27.605525 13762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:27.606952 13762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:27.607126 13762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:27.607275 13762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:27.611842 13767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.611828 13768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.611828 13770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:27.612151 13762 server_base.cc:1034] running on GCE node
I20260619 20:57:27.612371 13762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:27.612623 13762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:27.613796 13762 hybrid_clock.cc:648] HybridClock initialized: now 1781902647613786 us; error 22 us; skew 500 ppm
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:27.616127 13762 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:57:27.616511 13762 webserver.cc:460] Webserver started at http://127.0.98.61:37777/ using document root <none> and password file <none>
I20260619 20:57:27.616676 13762 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:27.616714 13762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:27.616793 13762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:27.617687 13762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data/instance:
uuid: "c30e264b2b354953b0fe62c7793e3b7b"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "f249ddbd1eb4ae201e39691f57296eb9"
server_key_iv: "b3c61503830eff23edda3dba57d844c7"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.617952 13762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal/instance:
uuid: "c30e264b2b354953b0fe62c7793e3b7b"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "f249ddbd1eb4ae201e39691f57296eb9"
server_key_iv: "b3c61503830eff23edda3dba57d844c7"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.619225 13762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:57:27.619957 13777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:27.620116 13762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:57:27.620208 13762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "c30e264b2b354953b0fe62c7793e3b7b"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "f249ddbd1eb4ae201e39691f57296eb9"
server_key_iv: "b3c61503830eff23edda3dba57d844c7"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.620286 13762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:27.632920 13755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } attempt: 2
I20260619 20:57:27.633301 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.633176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55826)
0619 20:57:27.633247 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.633274 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:57:27.633637 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.633527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55838)
0619 20:57:27.633589 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.633613 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 20:57:27.633715 13755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42817: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42817: connect: Connection refused (error 111)
I20260619 20:57:27.643860 13762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:27.646524 13762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:27.646663 13762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:27.651137 13762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42817
I20260619 20:57:27.651176 13829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42817 every 8 connection(s)
I20260619 20:57:27.651583 13762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:57:27.652487 13830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:27.653990 13830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:27.659839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13762
I20260619 20:57:27.659966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-1/wal/instance
I20260619 20:57:27.660143 392 external_mini_cluster.cc:1428] Setting key d863f797349e840a341343357d034493
I20260619 20:57:27.662868 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.654727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.0.98.62:46097)
0619 20:57:27.654793 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.654796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.655004 (+ 208us) server_negotiation.cc:407] Connection header received
0619 20:57:27.655557 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.655559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.655571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.655622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.656185 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.656651 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.657570 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.657782 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.657853 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.660022 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.660023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.660024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.660031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.661480 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.661918 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.661920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.661920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.661978 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.662234 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.662235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.662236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.662351 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.662430 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.662683 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.662760 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:57:27.663079 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.654714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46097 (local address 127.0.0.1:42380)
0619 20:57:27.654868 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.654917 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.655461 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.655695 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.655699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.655926 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.656116 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.656121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.656725 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.656725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.657479 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.657481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.657860 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.657861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.657960 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.658435 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.658443 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.659905 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.661591 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.661598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.661600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.661830 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.662048 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.662049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.662049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.662155 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.662422 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.662428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.662563 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.662666 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.662814 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260619 20:57:27.663621 13830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:57:27.673280 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.663881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55840)
0619 20:57:27.663952 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.663961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.663999 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.664239 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.664240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.664342 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.664386 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.664388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.665741 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.665742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.666277 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.666279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.666902 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.666903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.666950 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.667333 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.667337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.670612 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.672269 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.672270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.672270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.672414 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.672624 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.672625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.672625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.672673 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.672958 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.672959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.672994 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.673059 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.673169 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 20:57:27.673364 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.663956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.0.98.61:42817)
0619 20:57:27.664091 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.664096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.664116 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:57:27.664139 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.664142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.664156 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.664240 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.664454 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.665662 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.666351 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.666812 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.666892 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.670698 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.670705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.670705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.670711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.672185 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.672496 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.672499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.672499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.672554 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.672755 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.672756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.672757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.672891 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.672977 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.673113 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.673235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260619 20:57:27.674273 13830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:57:27.674605 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.674485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50806)
0619 20:57:27.674537 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.674579 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:57:27.674718 13778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
I20260619 20:57:27.675110 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.675017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50814)
0619 20:57:27.675065 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.675088 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:57:27.675208 13830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:57:27.682708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35647
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:27.724401 13830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } attempt: 1
I20260619 20:57:27.724875 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.724726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50824)
0619 20:57:27.724788 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.724838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:57:27.724885 13755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } attempt: 3
I20260619 20:57:27.725437 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.725310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50836)
0619 20:57:27.725372 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.725400 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:57:27.725591 13830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:57:27.732703 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.725314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.98.61:42817)
0619 20:57:27.725361 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.725364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.725394 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:57:27.725497 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.725497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.725509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.725573 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.726000 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.726587 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.727575 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.727785 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.727875 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.729870 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.729872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.729872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.729886 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.731440 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.731854 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.731856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.731856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.731931 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.732173 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.732174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.732175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.732290 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.732371 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.732536 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.732614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:57:27.732708 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.725277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55850)
0619 20:57:27.725337 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.725348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.725414 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.725649 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.725651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.725830 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.725907 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.725911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.726681 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.726682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.727484 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.727487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.727890 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.727891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.727970 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.728401 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.728407 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.729765 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.731573 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.731574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.731574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.731761 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.732007 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.732008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.732008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.732096 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.732363 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.732364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.732415 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.732511 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.732614 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:57:27.733347 13755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:57:27.733708 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.733606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50844)
0619 20:57:27.733656 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.733684 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:57:27.734025 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.733934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50856)
0619 20:57:27.733979 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.734004 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:57:27.734133 13755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
W20260619 20:57:27.764225 13837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:27.764411 13837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:27.764433 13837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 20:57:27.765275 13755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } attempt: 1
W20260619 20:57:27.765625 13837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:27.765676 13837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:27.765699 13837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:27.765717 13837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:27.765733 13837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:27.765746 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.765593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50862)
0619 20:57:27.765661 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.765708 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:57:27.766142 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.766021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50868)
0619 20:57:27.766078 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.766115 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:57:27.766263 13755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
I20260619 20:57:27.767313 13837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46097,127.0.98.61:42817,127.0.98.60:35647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:27.767493 13837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:27.767642 13837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:27.772097 13842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.772094 13843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:27.772125 13845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:27.772254 13837 server_base.cc:1034] running on GCE node
I20260619 20:57:27.772688 13837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:27.772984 13837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:27.774153 13837 hybrid_clock.cc:648] HybridClock initialized: now 1781902647774135 us; error 45 us; skew 500 ppm
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:27.776784 13837 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:57:27.777235 13837 webserver.cc:460] Webserver started at http://127.0.98.60:34331/ using document root <none> and password file <none>
I20260619 20:57:27.777487 13837 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:27.777559 13837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:27.777711 13837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:27.778829 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data/instance:
uuid: "c996917fa626498db204f45ddfe23a9e"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "20b084361f593a52d257c0f9d3255373"
server_key_iv: "c9164a7d774cbc1cffabff51a590f8ed"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.779203 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal/instance:
uuid: "c996917fa626498db204f45ddfe23a9e"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "20b084361f593a52d257c0f9d3255373"
server_key_iv: "c9164a7d774cbc1cffabff51a590f8ed"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.780602 13837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:57:27.781318 13852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:27.781488 13837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:27.781590 13837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "c996917fa626498db204f45ddfe23a9e"
format_stamp: "Formatted at 2026-06-19 20:57:27 on dist-test-slave-8bgz"
server_key: "20b084361f593a52d257c0f9d3255373"
server_key_iv: "c9164a7d774cbc1cffabff51a590f8ed"
server_key_version: "encryptionkey@0"
I20260619 20:57:27.781672 13837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:27.793748 13830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } attempt: 2
I20260619 20:57:27.794153 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.793998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50884)
0619 20:57:27.794069 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.794111 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:57:27.794553 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.794479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50890)
0619 20:57:27.794507 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.794533 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 20:57:27.794669 13830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35647: connect: Connection refused (error 111)
I20260619 20:57:27.795462 13837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:27.797516 13837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:27.797672 13837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:27.801362 13837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35647
I20260619 20:57:27.801405 13904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35647 every 8 connection(s)
I20260619 20:57:27.801725 13837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:57:27.802599 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:27.804044 13905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:57:27.806797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13837
I20260619 20:57:27.806901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902497118594-392-0/minicluster-data/master-2/wal/instance
I20260619 20:57:27.807108 392 external_mini_cluster.cc:1428] Setting key 0a9aae1c35731078f87dead3f90f7959
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:57:27.813328 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.804697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.0.98.62:46097)
0619 20:57:27.804758 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.804760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.804937 (+ 177us) server_negotiation.cc:407] Connection header received
0619 20:57:27.805407 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.805408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.805420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.805473 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.806038 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.806580 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.807872 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.808103 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.808157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.810316 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.810317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.810317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.810323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.811910 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.812363 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.812364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.812364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.812414 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.812706 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.812707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.812707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.812830 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.812904 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.813159 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.813242 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:57:27.813585 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.804638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46097 (local address 127.0.0.1:42384)
0619 20:57:27.804827 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.804857 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.805304 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.805542 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.805546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.805767 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.805943 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.805947 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.806718 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.806719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.807794 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.807797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.808187 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.808188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.808367 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.808879 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.808887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.810209 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.811987 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.811992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.811993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.812290 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.812479 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.812479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.812480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.812625 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.812908 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.812915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.813030 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.813137 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.813271 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
I20260619 20:57:27.814375 13905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:57:27.816035 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.807469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.0.98.60:35647)
0619 20:57:27.807647 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.807650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.807683 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:57:27.807757 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.807758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.807777 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.807872 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.808606 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.809360 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.810097 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.810540 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.810610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.813070 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.813077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.813078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.813085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.814858 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.815225 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.815227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.815228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.815287 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.815485 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.815486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.815486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.815620 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.815792 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.815833 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.815916 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:57:27.820895 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.814624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42817 (local address 127.0.0.1:55852)
0619 20:57:27.814660 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.814668 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.814698 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.814855 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.814856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.815004 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.815030 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.815031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.815620 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.815622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.816104 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.816106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.816474 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.816475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.816539 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.816890 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.816894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.818186 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.819990 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.819991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.819991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.820167 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.820361 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.820361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.820361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.820413 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.820702 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.820702 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.820733 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.820782 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.820841 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 20:57:27.820997 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.814693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.0.98.61:42817)
0619 20:57:27.814754 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.814757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.814764 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:57:27.814785 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.814786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.814795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.814845 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.815108 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.815549 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.816197 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.816428 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.816496 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.818318 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.818319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.818319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.818326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.819901 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.820237 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.820238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.820238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.820293 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.820472 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.820473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.820473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.820632 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.820707 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.820811 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.820885 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:57:27.821352 13905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:27.824815 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.816898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.0.98.62:46097)
0619 20:57:27.816960 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.816962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.816969 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:57:27.816987 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.816988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.816996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.817047 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.817614 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.818385 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.819419 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.819590 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.819637 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.822194 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.822196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.822196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.822231 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.823686 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.824039 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.824040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.824040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.824100 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.824289 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.824290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.824291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.824399 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.824604 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.824654 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.824724 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:57:27.828742 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.821639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.60:35647)
0619 20:57:27.821668 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.821670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.821679 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:57:27.821746 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.821747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.821756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.821797 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.822058 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.822788 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.823629 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.823927 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.824012 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.825810 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.825811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.825811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.825816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.827219 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.827918 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.827919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.827919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.827977 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.828204 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.828205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.828205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.828302 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.828360 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.828566 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.828661 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":16}
I20260619 20:57:27.828815 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.821590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50920)
0619 20:57:27.821638 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.821648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.821691 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.821809 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.821810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.821954 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.821977 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.821979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.822905 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.822907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.823560 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.823562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.824013 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.824014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.824061 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.824446 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.824450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.825734 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.827640 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.827641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.827641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.827850 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.828046 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.828047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.828048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.828122 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.828375 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.828376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.828433 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.828522 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.828620 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:57:27.830319 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: Bootstrap starting.
I20260619 20:57:27.831218 13905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:27.831506 13905 log.cc:824] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:27.832192 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: No bootstrap required, opened a new log
I20260619 20:57:27.832619 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.825497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.0.98.61:42817)
0619 20:57:27.825532 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.825534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.825541 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:57:27.825560 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.825561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.825569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.825627 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.826117 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.826638 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.827300 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.827503 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.827593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.830083 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.830084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.830085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.830091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.831514 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.831864 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.831865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.831865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.831918 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.832123 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.832124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.832124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.832225 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.832421 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.832459 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.832532 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:57:27.834010 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:57:27.834162 13905 raft_consensus.cc:348] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.834372 13905 raft_consensus.cc:374] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:27.834421 13905 raft_consensus.cc:729] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c996917fa626498db204f45ddfe23a9e, State: Initialized, Role: FOLLOWER
I20260619 20:57:27.834545 13905 consensus_queue.cc:260] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.834806 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:27.834863 13905 sys_catalog.cc:564] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:27.834882 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:57:27.836716 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:57:27.836802 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:57:27.836920 13837 master_runner.cc:186] Error getting master registration for 127.0.98.62:46097: OK, instance_id { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" instance_seqno: 1781902647498939 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:57:27.839232 13837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:57:27.841605 13755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } attempt: 2
I20260619 20:57:27.843781 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.835475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.0.98.62:46097)
0619 20:57:27.835514 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.835515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.835757 (+ 242us) server_negotiation.cc:407] Connection header received
0619 20:57:27.835854 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.835855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.835864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.835894 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.836676 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.837379 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.838389 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.838674 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.838712 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.840110 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.840111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.840112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.840118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.842247 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.842924 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.842925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.842925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.842983 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.843190 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.843191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.843191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.843308 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.843393 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.843619 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.843694 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:57:27.844439 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.835815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.0.98.60:35647)
0619 20:57:27.835849 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.835851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.836593 (+ 742us) server_negotiation.cc:407] Connection header received
0619 20:57:27.836699 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.836699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.836708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.836734 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.837312 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.838041 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.839052 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.839354 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.839398 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.840865 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.840866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.840867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.840873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.842857 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.843261 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.843262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.843262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.843319 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.843533 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.843534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.843534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.843698 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.843767 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.844254 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.844330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:57:27.846614 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.835519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.98.61:42817)
0619 20:57:27.835554 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.835556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.835725 (+ 169us) server_negotiation.cc:407] Connection header received
0619 20:57:27.835821 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.835822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.835832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.835861 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.840293 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.841011 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.841945 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.842264 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.842310 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.843699 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.843700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.843701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.843707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.845351 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.845775 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.845776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.845776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.845832 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.846026 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.846027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.846027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.846141 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.846227 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.846448 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.846527 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:57:27.851704 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.841794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50936)
0619 20:57:27.841877 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.841886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.841920 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.845222 (+ 3302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.845223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.845375 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.845413 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.845416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.846175 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.846176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.846931 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.846934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.847368 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.847369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.847418 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.847756 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.847759 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.849061 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.850625 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.850626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.850626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.850832 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.851098 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.851099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.851099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.851156 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.851411 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.851412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.851450 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.851533 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.851618 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 20:57:27.851703 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.842000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.98.60:35647)
0619 20:57:27.844506 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.844508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.844516 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:27.844549 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.844550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.844559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.844607 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.845480 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.846088 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.847008 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.847288 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.847367 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.849162 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.849163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.849164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.849171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.850535 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.850970 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.850971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.850971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.851028 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.851228 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.851229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.851230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.851333 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.851403 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.851559 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.851619 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2490,"thread_start_us":83,"threads_started":1}
I20260619 20:57:27.852939 13755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: Bootstrap starting.
I20260619 20:57:27.853552 13755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:27.853834 13755 log.cc:824] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:27.854580 13755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: No bootstrap required, opened a new log
I20260619 20:57:27.855969 13755 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.856082 13755 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:27.856104 13755 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0066e2b8cb4525bf64c4ec3046cb10, State: Initialized, Role: FOLLOWER
I20260619 20:57:27.856288 13755 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.856573 13755 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:27.856562 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:27.856657 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:57:27.858856 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:57:27.858948 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:57:27.859304 13687 master_runner.cc:186] Error getting master registration for 127.0.98.61:42817: OK, instance_id { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" instance_seqno: 1781902647649792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:57:27.861570 13687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:57:27.885859 13830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } attempt: 3
Jun 19 20:57:27 dist-test-slave-8bgz krb5kdc[13674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:57:27.893810 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.886094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35647 (local address 127.0.0.1:50938)
0619 20:57:27.886157 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:27.886169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:27.886204 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:27.886423 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:27.886424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:27.886608 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:27.886654 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.886655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.887584 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.887586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.888410 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:27.888412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.888861 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.888862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:27.888919 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.889441 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:27.889446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:27.890978 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:27.892743 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.892744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.892745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.892949 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.893147 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:27.893148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:27.893148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:27.893221 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:27.893509 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:27.893510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:27.893552 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:57:27.893623 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:57:27.893712 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:57:27.893815 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:27.886182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.0.98.60:35647)
0619 20:57:27.886295 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:27.886297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:27.886303 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:57:27.886322 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:27.886323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:27.886331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:27.886374 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:27.886733 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.887490 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.888498 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:27.888786 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:27.888856 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:27.891072 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:27.891073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:27.891074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:27.891080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:27.892660 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.893023 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.893024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.893024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.893079 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:27.893299 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:27.893300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:27.893314 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:27.893428 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:27.893508 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:27.893656 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 20:57:27.893726 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 20:57:27.895681 13830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: Bootstrap starting.
I20260619 20:57:27.896380 13830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:27.896682 13830 log.cc:824] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:27.897527 13830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: No bootstrap required, opened a new log
I20260619 20:57:27.899448 13830 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.899608 13830 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:27.899643 13830 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30e264b2b354953b0fe62c7793e3b7b, State: Initialized, Role: FOLLOWER
I20260619 20:57:27.899767 13830 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:27.900044 13830 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:27.900003 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:27.900108 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:57:27.901893 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:57:27.901990 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:57:27.903769 13762 master_runner.cc:263] No leader master found from master c30e264b2b354953b0fe62c7793e3b7b
I20260619 20:57:27.903810 13762 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 20:57:27.996667 13708 tablet.cc:2369] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:28.044463 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:28.044648 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:28.044958 13947 leader_election.cc:288] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee0066e2b8cb4525bf64c4ec3046cb10 (127.0.98.62:46097), c996917fa626498db204f45ddfe23a9e (127.0.98.60:35647)
I20260619 20:57:28.045226 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c30e264b2b354953b0fe62c7793e3b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c996917fa626498db204f45ddfe23a9e" is_pre_election: true
I20260619 20:57:28.045210 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c30e264b2b354953b0fe62c7793e3b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" is_pre_election: true
I20260619 20:57:28.045382 13729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30e264b2b354953b0fe62c7793e3b7b in term 0.
I20260619 20:57:28.045383 13879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30e264b2b354953b0fe62c7793e3b7b in term 0.
I20260619 20:57:28.045594 13778 leader_election.cc:302] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30e264b2b354953b0fe62c7793e3b7b, c996917fa626498db204f45ddfe23a9e; no voters:
I20260619 20:57:28.045744 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:57:28.045821 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:57:28.045847 13947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:28.046802 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:28.046958 13947 leader_election.cc:288] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [CANDIDATE]: Term 1 election: Requested vote from peers ee0066e2b8cb4525bf64c4ec3046cb10 (127.0.98.62:46097), c996917fa626498db204f45ddfe23a9e (127.0.98.60:35647)
I20260619 20:57:28.047108 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c30e264b2b354953b0fe62c7793e3b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c996917fa626498db204f45ddfe23a9e"
I20260619 20:57:28.047123 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c30e264b2b354953b0fe62c7793e3b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10"
I20260619 20:57:28.047194 13879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:28.047211 13729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:28.047881 13729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30e264b2b354953b0fe62c7793e3b7b in term 1.
I20260619 20:57:28.048036 13879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30e264b2b354953b0fe62c7793e3b7b in term 1.
I20260619 20:57:28.048084 13781 leader_election.cc:302] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30e264b2b354953b0fe62c7793e3b7b, ee0066e2b8cb4525bf64c4ec3046cb10; no voters:
W20260619 20:57:28.048166 13858 tablet.cc:2369] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:28.048194 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:28.048350 13947 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [term 1 LEADER]: Becoming Leader. State: Replica: c30e264b2b354953b0fe62c7793e3b7b, State: Running, Role: LEADER
I20260619 20:57:28.048448 13947 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } }
I20260619 20:57:28.048810 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30e264b2b354953b0fe62c7793e3b7b. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.048897 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:28.049091 13962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:28.049278 13962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:28.050882 13879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [term 1 FOLLOWER]: Refusing update from remote peer c30e264b2b354953b0fe62c7793e3b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:28.050969 13729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [term 1 FOLLOWER]: Refusing update from remote peer c30e264b2b354953b0fe62c7793e3b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:28.051183 13960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:28.051290 13947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:28.052456 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30e264b2b354953b0fe62c7793e3b7b. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.052454 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30e264b2b354953b0fe62c7793e3b7b. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.052548 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:57:28.052546 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:57:28.053984 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.054080 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:28.054235 13964 mvcc.cc:205] Tried to move back new op lower bound from 7298673246414196736 to 7298673246407028736. Current Snapshot: MvccSnapshot[applied={T|T < 7298673246414196736}]
I20260619 20:57:28.054181 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.054344 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:57:28.054374 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.054451 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:57:28.053982 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c30e264b2b354953b0fe62c7793e3b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0066e2b8cb4525bf64c4ec3046cb10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46097 } } peers { permanent_uuid: "c30e264b2b354953b0fe62c7793e3b7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42817 } } peers { permanent_uuid: "c996917fa626498db204f45ddfe23a9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35647 } } }
I20260619 20:57:28.054585 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:28.054658 13965 mvcc.cc:205] Tried to move back new op lower bound from 7298673246414196736 to 7298673246407028736. Current Snapshot: MvccSnapshot[applied={T|T < 7298673246414196736}]
I20260619 20:57:28.054803 13963 mvcc.cc:205] Tried to move back new op lower bound from 7298673246414196736 to 7298673246407028736. Current Snapshot: MvccSnapshot[applied={T|T < 7298673246414196736}]
I20260619 20:57:28.055068 13962 catalog_manager.cc:1347] Generated new cluster ID: 5e2b19ba05414cf1a04f80d08dcb5929
I20260619 20:57:28.055117 13962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:57:28.091107 13962 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:57:28.091694 13962 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:28.095890 13962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30e264b2b354953b0fe62c7793e3b7b: Generated new TSK 0
I20260619 20:57:28.096081 13962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:57:28.837239 13929 catalog_manager.cc:1259] Loaded cluster ID: 5e2b19ba05414cf1a04f80d08dcb5929
I20260619 20:57:28.837330 13929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: loading cluster ID for follower catalog manager: success
I20260619 20:57:28.838094 13929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: acquiring CA information for follower catalog manager: success
I20260619 20:57:28.838421 13929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c996917fa626498db204f45ddfe23a9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:57:28.859375 13944 catalog_manager.cc:1259] Loaded cluster ID: 5e2b19ba05414cf1a04f80d08dcb5929
I20260619 20:57:28.859443 13944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: loading cluster ID for follower catalog manager: success
I20260619 20:57:28.860257 13944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: acquiring CA information for follower catalog manager: success
I20260619 20:57:28.860580 13944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee0066e2b8cb4525bf64c4ec3046cb10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:57:29.139721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13687
I20260619 20:57:29.144784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13762
I20260619 20:57:29.149622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13837
2026-06-19T20:57:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2899 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:57:29 dist-test-slave-8bgz krb5kdc[13977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:57:29 dist-test-slave-8bgz krb5kdc[13977](info): set up 2 sockets
Jun 19 20:57:29 dist-test-slave-8bgz krb5kdc[13977](info): commencing operation
krb5kdc: starting...
W20260619 20:57:30.199719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:57:30.322158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:30.404609 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:30.404816 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:30.404840 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:30.405978 13990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:30.406023 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:30.406039 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:30.406054 13990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:30.406067 13990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:30.407559 13990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:30.407755 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:30.407946 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:30.412585 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:30.412585 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:30.412644 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:30.412773 13990 server_base.cc:1034] running on GCE node
I20260619 20:57:30.413095 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:30.413393 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:30.414556 13990 hybrid_clock.cc:648] HybridClock initialized: now 1781902650414544 us; error 39 us; skew 500 ppm
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:30.419776 13990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:30.420260 13990 webserver.cc:460] Webserver started at http://127.0.98.62:34539/ using document root <none> and password file <none>
I20260619 20:57:30.420513 13990 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:30.420588 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:30.420753 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:30.421784 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "21075acbac4f42b8b66b7500292a4e22"
format_stamp: "Formatted at 2026-06-19 20:57:30 on dist-test-slave-8bgz"
server_key: "697a58f7296218953602e3cca4691c31"
server_key_iv: "463b5919b98b8eac0875a417dfcb125d"
server_key_version: "encryptionkey@0"
I20260619 20:57:30.422155 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "21075acbac4f42b8b66b7500292a4e22"
format_stamp: "Formatted at 2026-06-19 20:57:30 on dist-test-slave-8bgz"
server_key: "697a58f7296218953602e3cca4691c31"
server_key_iv: "463b5919b98b8eac0875a417dfcb125d"
server_key_version: "encryptionkey@0"
I20260619 20:57:30.423547 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:30.424306 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:30.424501 13990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:30.424611 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "21075acbac4f42b8b66b7500292a4e22"
format_stamp: "Formatted at 2026-06-19 20:57:30 on dist-test-slave-8bgz"
server_key: "697a58f7296218953602e3cca4691c31"
server_key_iv: "463b5919b98b8eac0875a417dfcb125d"
server_key_version: "encryptionkey@0"
I20260619 20:57:30.424687 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:30.450552 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:30.452518 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:30.452685 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:30.456830 13990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34721
I20260619 20:57:30.456867 14057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34721 every 8 connection(s)
I20260619 20:57:30.457212 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:30.458142 14058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:30.461570 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22: Bootstrap starting.
I20260619 20:57:30.462221 14058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:30.462589 14058 log.cc:824] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:30.463409 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22: No bootstrap required, opened a new log
I20260619 20:57:30.465144 14058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260619 20:57:30.465315 14058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:30.465381 14058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21075acbac4f42b8b66b7500292a4e22, State: Initialized, Role: FOLLOWER
I20260619 20:57:30.465539 14058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260619 20:57:30.465628 14058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:30.465677 14058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:30.465744 14058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:30.466302 14058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260619 20:57:30.466426 14058 leader_election.cc:302] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21075acbac4f42b8b66b7500292a4e22; no voters:
I20260619 20:57:30.466620 14058 leader_election.cc:288] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:57:30.466639 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:30.466749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 13990
I20260619 20:57:30.466806 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [term 1 LEADER]: Becoming Leader. State: Replica: 21075acbac4f42b8b66b7500292a4e22, State: Running, Role: LEADER
I20260619 20:57:30.466851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:57:30.466883 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } }
I20260619 20:57:30.466925 14058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:30.467095 392 external_mini_cluster.cc:1428] Setting key 435072dd034832bf1c28c9e68e43361b
I20260619 20:57:30.467152 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21075acbac4f42b8b66b7500292a4e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } } }
I20260619 20:57:30.467217 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:30.467478 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21075acbac4f42b8b66b7500292a4e22. Latest consensus state: current_term: 1 leader_uuid: "21075acbac4f42b8b66b7500292a4e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21075acbac4f42b8b66b7500292a4e22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34721 } } }
I20260619 20:57:30.467604 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:30.468118 14065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:30.468632 14065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:30.470597 14065 catalog_manager.cc:1347] Generated new cluster ID: 394cfbb40cc2450a87352b03d31b9c62
I20260619 20:57:30.470646 14065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[13977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:30.480357 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:30.467784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.98.62:34721)
0619 20:57:30.468920 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:30.468929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:30.468966 (+ 37us) server_negotiation.cc:407] Connection header received
0619 20:57:30.469244 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:30.469252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:30.469319 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:30.469485 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:30.470280 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:30.471833 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:30.472896 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:30.473586 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:30.473709 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:30.476688 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:30.476691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:30.476692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:30.476703 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:30.478676 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:30.479105 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:30.479108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:30.479108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:30.479216 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:30.479446 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:30.479447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:30.479447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:30.479724 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:30.479866 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:30.480015 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 20:57:30.480141 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":257,"thread_start_us":83,"threads_started":1}
I20260619 20:57:30.481308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:57:30.491094 14065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:57:30.491607 14065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:30.504194 14065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21075acbac4f42b8b66b7500292a4e22: Generated new TSK 0
I20260619 20:57:30.504400 14065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:57:30.535405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 13990
2026-06-19T20:57:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1384 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[14085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[14085](info): set up 2 sockets
Jun 19 20:57:30 dist-test-slave-8bgz krb5kdc[14085](info): commencing operation
krb5kdc: starting...
W20260619 20:57:31.562417 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:57:31 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:57:31.673028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:31.750823 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:31.751046 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:31.751081 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:31.752275 14098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:31.752344 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:31.752367 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:31.752388 14098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:31.752416 14098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:31.753832 14098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:31.754096 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:31.754338 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:31.759028 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:31.758991 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:31.759203 14098 server_base.cc:1034] running on GCE node
W20260619 20:57:31.758986 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:31.759572 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:31.759867 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:31.761008 14098 hybrid_clock.cc:648] HybridClock initialized: now 1781902651760986 us; error 34 us; skew 500 ppm
Jun 19 20:57:31 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:31.763386 14098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:31.763840 14098 webserver.cc:460] Webserver started at http://127.0.98.62:39605/ using document root <none> and password file <none>
I20260619 20:57:31.764048 14098 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:31.764092 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:31.764253 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:31.765269 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "cf5d155aaeb8483388ac9d2f228abc39"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "c096dd89f5aa7a72c4cd10cdd139ae6c"
server_key_iv: "bc7b11f8e92cd3c382ff999dd969c19b"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.765614 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf5d155aaeb8483388ac9d2f228abc39"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "c096dd89f5aa7a72c4cd10cdd139ae6c"
server_key_iv: "bc7b11f8e92cd3c382ff999dd969c19b"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.766959 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:57:31.767634 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:31.767807 14098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:31.767874 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "cf5d155aaeb8483388ac9d2f228abc39"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "c096dd89f5aa7a72c4cd10cdd139ae6c"
server_key_iv: "bc7b11f8e92cd3c382ff999dd969c19b"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.767961 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:31.774899 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:31.776502 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:31.776651 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:31.780537 14098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36083
I20260619 20:57:31.780565 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36083 every 8 connection(s)
I20260619 20:57:31.780900 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:31.781778 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:31.783979 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap starting.
I20260619 20:57:31.784520 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:31.784794 14166 log.cc:824] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:31.785497 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: No bootstrap required, opened a new log
I20260619 20:57:31.787115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14098
I20260619 20:57:31.787237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:57:31.787444 392 external_mini_cluster.cc:1428] Setting key eabcf7a3df805058eee73ae7fb138446
I20260619 20:57:31.787534 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:31.787734 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:31.787786 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Initialized, Role: FOLLOWER
I20260619 20:57:31.787940 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:31.788072 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:31.788179 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:31.788239 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:31.789112 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:31.789248 14166 leader_election.cc:302] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5d155aaeb8483388ac9d2f228abc39; no voters:
I20260619 20:57:31.789429 14166 leader_election.cc:288] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:57:31.789469 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:31.789642 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 LEADER]: Becoming Leader. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Running, Role: LEADER
I20260619 20:57:31.789718 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:31.789783 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:31.790138 14171 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5d155aaeb8483388ac9d2f228abc39. Latest consensus state: current_term: 1 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:31.790351 14171 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:31.790284 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:31.790556 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:31.790987 14180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:31.791612 14180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:31.793275 14180 catalog_manager.cc:1347] Generated new cluster ID: cc188e1f554d410b8d445560fd86cd09
I20260619 20:57:31.793329 14180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:57:31 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:31.798517 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:31.788002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.62:36083)
0619 20:57:31.788191 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:31.788200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:31.788255 (+ 55us) server_negotiation.cc:407] Connection header received
0619 20:57:31.788373 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:31.788378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:31.788425 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:31.788547 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:31.789194 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.790582 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.791718 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.792154 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.792288 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:31.795058 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:31.795062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:31.795062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:31.795074 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:31.796867 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:31.797270 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:31.797273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:31.797274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:31.797381 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:31.797603 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:31.797604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:31.797604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:31.797867 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:31.798061 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:31.798129 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:57:31.798308 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260619 20:57:31.805594 14180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:57:31.806084 14180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:31.808593 14180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Generated new TSK 0
I20260619 20:57:31.808758 14180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:57:31.834780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:31.920336 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:31.920526 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:31.920548 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:31.921659 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:31.921698 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:31.921725 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:31.923173 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:31.923396 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:31.923626 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:31.929390 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:31.929527 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:31.929615 14192 server_base.cc:1034] running on GCE node
W20260619 20:57:31.929494 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:31.929889 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:31.930162 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:31.931361 14192 hybrid_clock.cc:648] HybridClock initialized: now 1781902651931347 us; error 35 us; skew 500 ppm
Jun 19 20:57:31 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:31.934087 14192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:31.934589 14192 webserver.cc:460] Webserver started at http://127.0.98.1:35051/ using document root <none> and password file <none>
I20260619 20:57:31.934852 14192 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:31.934896 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:31.935034 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:31.936045 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4c9fed3d38242df9a36b548c8e7641c"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "f00fb8ed6070133af199d39a5d1c9d7a"
server_key_iv: "a9a1185a03d8ef4c74c7c16c3f74a87a"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.936386 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c9fed3d38242df9a36b548c8e7641c"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "f00fb8ed6070133af199d39a5d1c9d7a"
server_key_iv: "a9a1185a03d8ef4c74c7c16c3f74a87a"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.937670 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:57:31.938437 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:31.938632 14192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:57:31.938730 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b4c9fed3d38242df9a36b548c8e7641c"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "f00fb8ed6070133af199d39a5d1c9d7a"
server_key_iv: "a9a1185a03d8ef4c74c7c16c3f74a87a"
server_key_version: "encryptionkey@0"
I20260619 20:57:31.938807 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:31.954489 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:31.956684 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:31.956837 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:31.957051 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:31.957374 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:31.957427 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:31.957465 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:31.957494 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:31.963565 14192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40463
I20260619 20:57:31.963619 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40463 every 8 connection(s)
I20260619 20:57:31.963948 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:31.969559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14192
I20260619 20:57:31.969689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:57:31.969878 392 external_mini_cluster.cc:1428] Setting key da2592c74a5a3910dbb3f9b07736b750
Jun 19 20:57:31 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:31.973577 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:31.964608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:34893)
0619 20:57:31.964771 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:31.964804 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:31.965293 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:31.965538 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:31.965541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:31.966086 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:31.966337 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:31.966342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.967031 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.967032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:31.967845 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:31.967846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.968224 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.968225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:31.968348 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:31.968771 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:31.968779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:31.970434 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:31.972353 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:31.972359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:31.972360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:31.972564 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:31.972781 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:31.972781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:31.972781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:31.972871 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:31.973144 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:31.973148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:31.973249 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:57:31.973326 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:57:31.973435 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260619 20:57:31.973599 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:31.964595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34893 (local address 127.0.98.62:36083)
0619 20:57:31.964636 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:31.964638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:31.964883 (+ 245us) server_negotiation.cc:407] Connection header received
0619 20:57:31.965392 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:31.965394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:31.965409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:31.965471 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:31.966427 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.966934 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.967940 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:31.968135 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:31.968215 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:31.970553 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:31.970555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:31.970555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:31.970563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:31.972270 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:31.972638 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:31.972639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:31.972640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:31.972704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:31.972934 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:31.972935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:31.972935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:31.973080 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:31.973148 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:31.973373 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 20:57:31.973464 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:57:31.974017 14321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:31.974113 14321 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:31.974327 14321 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:31.974864 14130 ts_manager.cc:194] Registered new tserver with Master: b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463)
I20260619 20:57:31.975493 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34893
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:57:31.994058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:32.073305 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.073523 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.073561 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.074723 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.074784 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.074829 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:32.076231 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.076524 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.076748 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:57:32.081656 14326 server_base.cc:1034] running on GCE node
W20260619 20:57:32.081624 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.081597 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.081625 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.082038 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.082350 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.083492 14326 hybrid_clock.cc:648] HybridClock initialized: now 1781902652083480 us; error 29 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.085932 14326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:32.086431 14326 webserver.cc:460] Webserver started at http://127.0.98.2:35303/ using document root <none> and password file <none>
I20260619 20:57:32.086647 14326 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.086714 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.086864 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:32.087834 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "14032d24e84541289b3e0008d3bb8bfe"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "aa95ae905b7be3ff876c8f28c1bb6778"
server_key_iv: "2b6e7b60e033a2765e20384134e03677"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.088189 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14032d24e84541289b3e0008d3bb8bfe"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "aa95ae905b7be3ff876c8f28c1bb6778"
server_key_iv: "2b6e7b60e033a2765e20384134e03677"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.089430 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:57:32.090184 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.090402 14326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.090500 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "14032d24e84541289b3e0008d3bb8bfe"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "aa95ae905b7be3ff876c8f28c1bb6778"
server_key_iv: "2b6e7b60e033a2765e20384134e03677"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.090590 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:32.110934 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:32.113212 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:32.113373 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:32.113592 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:32.113894 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:32.113947 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.113986 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:32.114014 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.119863 14326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260619 20:57:32.119912 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260619 20:57:32.120218 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:32.128981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14326
I20260619 20:57:32.129083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:57:32.129232 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.120902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57603 (local address 127.0.98.62:36083)
0619 20:57:32.120964 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.120966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.121167 (+ 201us) server_negotiation.cc:407] Connection header received
0619 20:57:32.121656 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.121658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.121671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.121742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.122648 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.123167 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.124117 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.124343 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.124407 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.126573 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.126574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.126575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.126580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.127995 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.128329 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.128330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.128330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.128379 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.128578 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.128579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.128579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.128686 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.128737 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.129058 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.129159 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:57:32.129280 392 external_mini_cluster.cc:1428] Setting key 80bf84ba7151c9d5ad46a502eb914d52
I20260619 20:57:32.129436 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.120878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:57603)
0619 20:57:32.121037 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.121076 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.121541 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.121813 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.121816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.122382 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.122559 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.122564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.123248 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.123249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.124022 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.124024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.124420 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.124421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.124516 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.124969 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.124979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.126490 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.128060 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.128065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.128066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.128259 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.128436 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.128437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.128437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.128524 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.128741 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.128745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.128901 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.129020 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.129192 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260619 20:57:32.130059 14455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:32.130170 14455 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:32.130368 14455 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:32.130795 14130 ts_manager.cc:194] Registered new tserver with Master: 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
I20260619 20:57:32.131177 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57603
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:57:32.153396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:32.233027 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.233208 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.233229 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.234436 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.234474 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.234500 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:32.235931 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.236163 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.236416 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:32.241325 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.241303 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.241542 14460 server_base.cc:1034] running on GCE node
W20260619 20:57:32.241320 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.241837 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.242075 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.243245 14460 hybrid_clock.cc:648] HybridClock initialized: now 1781902652243221 us; error 36 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.245594 14460 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:32.246047 14460 webserver.cc:460] Webserver started at http://127.0.98.3:37189/ using document root <none> and password file <none>
I20260619 20:57:32.246344 14460 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.246410 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.246551 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:32.247573 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2d987308c4341da87f4f7d44e35fad8"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "01bb7fc49532859d9ea4c65b58ad6c2e"
server_key_iv: "00853cf8bb1e9d290848e8c0b4a69c51"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.247923 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d987308c4341da87f4f7d44e35fad8"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "01bb7fc49532859d9ea4c65b58ad6c2e"
server_key_iv: "00853cf8bb1e9d290848e8c0b4a69c51"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.249221 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:57:32.249922 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.250090 14460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:57:32.250195 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "b2d987308c4341da87f4f7d44e35fad8"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "01bb7fc49532859d9ea4c65b58ad6c2e"
server_key_iv: "00853cf8bb1e9d290848e8c0b4a69c51"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.250283 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:32.275846 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:32.277846 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:32.278012 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:32.278308 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:32.278577 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:32.278609 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.278633 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:32.278645 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.284425 14460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43673
I20260619 20:57:32.284546 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43673 every 8 connection(s)
I20260619 20:57:32.284816 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:32.287796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14460
I20260619 20:57:32.287879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:57:32.288103 392 external_mini_cluster.cc:1428] Setting key 2b9155eebf18afb7b48eec7172874604
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:32.294615 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.285527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47223 (local address 127.0.98.62:36083)
0619 20:57:32.285611 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.285613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.285793 (+ 180us) server_negotiation.cc:407] Connection header received
0619 20:57:32.286371 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.286373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.286384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.286436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.287377 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.287947 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.289269 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.289537 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.289623 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.291948 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.291949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.291950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.291958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.293188 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.293582 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.293583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.293584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.293646 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.293919 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.293920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.293920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.294037 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.294109 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.294448 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.294525 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:57:32.294821 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.285493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:47223)
0619 20:57:32.285665 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.285711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.286262 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.286506 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.286510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.287083 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.287268 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.287273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.288027 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.288028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.289178 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.289182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.289638 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.289639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.289770 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.290326 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.290335 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.291854 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.293279 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.293286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.293288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.293507 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.293713 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.293714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.293714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.293825 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.294130 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.294136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.294321 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.294426 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.294577 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260619 20:57:32.295285 14589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:32.295362 14589 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:32.295547 14589 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:32.295912 14130 ts_manager.cc:194] Registered new tserver with Master: b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673)
I20260619 20:57:32.296232 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47223
I20260619 20:57:32.299819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:57:32.306370 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.301081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.0.98.62:36083)
0619 20:57:32.301141 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.301143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.301187 (+ 44us) server_negotiation.cc:407] Connection header received
0619 20:57:32.301245 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.301245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.301253 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.301301 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.301800 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.302302 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.302965 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.303136 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.303183 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.304204 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.304205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.304205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.304211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.305363 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.305678 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.305678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.305679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.305728 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.305888 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.305889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.305889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.306018 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.306089 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.306186 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.306311 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:57:32.307734 14130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:57:32.309479 14130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:57:32.315619 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.311626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:49562)
0619 20:57:32.311955 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.311967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.312024 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.312354 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.312357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.312360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:32.312434 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.312437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313340 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.313341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.314370 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.314373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315304 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.315382 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.315395 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.315494 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.315548 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260619 20:57:32.315629 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.311860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.98.2:39351)
0619 20:57:32.312141 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.312145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.312169 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:57:32.312208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.312213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.312226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.312441 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:32.312526 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313226 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.314465 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315214 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315291 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.315354 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.315513 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.315557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":69,"threads_started":1}
I20260619 20:57:32.316023 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.311836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43673 (local address 127.0.0.1:38752)
0619 20:57:32.312040 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.312047 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.312080 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.312628 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.312629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.312631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:32.312664 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.312665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313931 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.313932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.314827 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.314830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315785 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.315856 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.315861 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.315909 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.315949 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260619 20:57:32.316099 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.311626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40463 (local address 127.0.0.1:57722)
0619 20:57:32.311796 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.311832 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.311895 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.312387 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.312388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.312390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:32.312441 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.312443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313886 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.313887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.314985 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.314987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315873 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.315935 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.315938 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.315966 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.315996 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260619 20:57:32.316324 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.312028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.0.98.3:43673)
0619 20:57:32.312200 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.312204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.312214 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:57:32.312254 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.312257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.312282 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.312361 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:32.312764 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313832 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.314911 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315617 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315687 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.315736 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.316195 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.316240 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260619 20:57:32.316474 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.311769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.1:40463)
0619 20:57:32.312217 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.312221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.312231 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:57:32.312268 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.312270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.312284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.312531 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:32.312608 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.313762 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.315068 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.315785 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.316224 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.316316 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.316342 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.316392 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":100,"threads_started":1}
I20260619 20:57:32.316521 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (DEFAULT_TABLE table=test-table [id=1167b2d2a0e94c158909cadbfd9575b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:32.316928 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:32.316934 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (DEFAULT_TABLE table=test-table [id=1167b2d2a0e94c158909cadbfd9575b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:32.317166 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:32.317114 14255 tablet_service.cc:1467] Processing CreateTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (DEFAULT_TABLE table=test-table [id=1167b2d2a0e94c158909cadbfd9575b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:32.317411 14255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:32.319586 14606 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Bootstrap starting.
I20260619 20:57:32.320199 14606 tablet_bootstrap.cc:654] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:32.320245 14607 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Bootstrap starting.
I20260619 20:57:32.320685 14608 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap starting.
I20260619 20:57:32.320843 14606 log.cc:824] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.320943 14607 tablet_bootstrap.cc:654] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:32.321308 14608 tablet_bootstrap.cc:654] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:32.321476 14607 log.cc:824] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.321810 14608 log.cc:824] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.322104 14607 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: No bootstrap required, opened a new log
I20260619 20:57:32.322160 14607 ts_tablet_manager.cc:1397] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:57:32.322569 14608 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: No bootstrap required, opened a new log
I20260619 20:57:32.322623 14608 ts_tablet_manager.cc:1397] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:32.323563 14607 raft_consensus.cc:348] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.323719 14607 raft_consensus.cc:374] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:32.323772 14607 raft_consensus.cc:729] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14032d24e84541289b3e0008d3bb8bfe, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.323905 14607 consensus_queue.cc:260] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.324039 14608 raft_consensus.cc:348] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.324180 14607 ts_tablet_manager.cc:1428] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:57:32.324214 14455 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
I20260619 20:57:32.324220 14608 raft_consensus.cc:374] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:32.324261 14608 raft_consensus.cc:729] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c9fed3d38242df9a36b548c8e7641c, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.324361 14608 consensus_queue.cc:260] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.324609 14608 ts_tablet_manager.cc:1428] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:57:32.324636 14606 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: No bootstrap required, opened a new log
I20260619 20:57:32.324698 14606 ts_tablet_manager.cc:1397] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 20:57:32.324934 14321 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
I20260619 20:57:32.326623 14606 raft_consensus.cc:348] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.326766 14606 raft_consensus.cc:374] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:32.326812 14606 raft_consensus.cc:729] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d987308c4341da87f4f7d44e35fad8, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.326911 14606 consensus_queue.cc:260] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.327162 14606 ts_tablet_manager.cc:1428] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:57:32.327251 14589 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
W20260619 20:57:32.370731 14456 tablet.cc:2369] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:32.401795 14613 raft_consensus.cc:482] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:32.401912 14613 raft_consensus.cc:504] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.402277 14613 leader_election.cc:288] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351), b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673)
I20260619 20:57:32.405072 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.402444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.1:38629)
0619 20:57:32.402494 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.402506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.402549 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.402730 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.402731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.402733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:32.402811 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.402812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.403447 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.403448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.404187 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.404189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.404803 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.404804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.404880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.404884 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.404941 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.404976 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:57:32.405081 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.402523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38629 (local address 127.0.98.2:39351)
0619 20:57:32.402578 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.402581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.402592 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:57:32.402621 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.402622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.402635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.402725 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:32.402884 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.403360 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.404285 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.404726 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.404804 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.404815 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.404954 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.405002 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:57:32.405340 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.402561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43673 (local address 127.0.98.1:53277)
0619 20:57:32.402708 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.402718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.402772 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.402903 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.402904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.402905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:32.402981 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.402982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.403589 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.403590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.404345 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.404346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.405072 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.405073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.405172 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.405177 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.405233 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.405270 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":8,"thread_start_us":105,"threads_started":1}
I20260619 20:57:32.405357 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.402729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53277 (local address 127.0.98.3:43673)
0619 20:57:32.402773 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.402775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.402784 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:57:32.402835 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.402836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.402846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.402900 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:32.403057 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.403511 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.404433 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.405000 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.405062 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.405070 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.405254 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.405294 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:57:32.405541 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6329a3d3cd0b4afc9f435c9f74eb6d32" candidate_uuid: "b4c9fed3d38242df9a36b548c8e7641c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14032d24e84541289b3e0008d3bb8bfe" is_pre_election: true
I20260619 20:57:32.405700 14409 raft_consensus.cc:2454] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c9fed3d38242df9a36b548c8e7641c in term 0.
I20260619 20:57:32.405745 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6329a3d3cd0b4afc9f435c9f74eb6d32" candidate_uuid: "b4c9fed3d38242df9a36b548c8e7641c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d987308c4341da87f4f7d44e35fad8" is_pre_election: true
I20260619 20:57:32.405887 14543 raft_consensus.cc:2454] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c9fed3d38242df9a36b548c8e7641c in term 0.
I20260619 20:57:32.405933 14208 leader_election.cc:302] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14032d24e84541289b3e0008d3bb8bfe, b4c9fed3d38242df9a36b548c8e7641c; no voters:
I20260619 20:57:32.406109 14613 raft_consensus.cc:2781] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:57:32.406165 14613 raft_consensus.cc:482] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:57:32.406213 14613 raft_consensus.cc:3037] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:32.406888 14613 raft_consensus.cc:504] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.407004 14613 leader_election.cc:288] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 1 election: Requested vote from peers 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351), b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673)
I20260619 20:57:32.407155 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6329a3d3cd0b4afc9f435c9f74eb6d32" candidate_uuid: "b4c9fed3d38242df9a36b548c8e7641c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14032d24e84541289b3e0008d3bb8bfe"
I20260619 20:57:32.407189 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6329a3d3cd0b4afc9f435c9f74eb6d32" candidate_uuid: "b4c9fed3d38242df9a36b548c8e7641c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d987308c4341da87f4f7d44e35fad8"
I20260619 20:57:32.407232 14409 raft_consensus.cc:3037] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:32.407253 14543 raft_consensus.cc:3037] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:32.408089 14543 raft_consensus.cc:2454] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c9fed3d38242df9a36b548c8e7641c in term 1.
I20260619 20:57:32.408088 14409 raft_consensus.cc:2454] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c9fed3d38242df9a36b548c8e7641c in term 1.
I20260619 20:57:32.408239 14208 leader_election.cc:302] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d987308c4341da87f4f7d44e35fad8, b4c9fed3d38242df9a36b548c8e7641c; no voters:
I20260619 20:57:32.408339 14613 raft_consensus.cc:2781] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:32.408475 14613 raft_consensus.cc:686] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 LEADER]: Becoming Leader. State: Replica: b4c9fed3d38242df9a36b548c8e7641c, State: Running, Role: LEADER
I20260619 20:57:32.408541 14613 consensus_queue.cc:237] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.409322 14129 catalog_manager.cc:5495] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b4c9fed3d38242df9a36b548c8e7641c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } health_report { overall_health: HEALTHY } } }
I20260619 20:57:32.432304 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.430110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.1:40463)
0619 20:57:32.430169 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.430171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.430179 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:32.430230 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.430231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.430249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.430314 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:57:32.430428 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.430975 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.431692 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.431875 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.431921 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.432047 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:57:32.432140 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:57:32.432197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.432234 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.432250 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:57:32.434166 14543 raft_consensus.cc:1261] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Refusing update from remote peer b4c9fed3d38242df9a36b548c8e7641c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:32.434166 14409 raft_consensus.cc:1261] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Refusing update from remote peer b4c9fed3d38242df9a36b548c8e7641c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:32.434551 14616 consensus_queue.cc:1035] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:32.434723 14613 consensus_queue.cc:1035] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:32.437737 14619 mvcc.cc:205] Tried to move back new op lower bound from 7298673264367345664 to 7298673264266018816. Current Snapshot: MvccSnapshot[applied={T|T < 7298673264367345664}]
I20260619 20:57:32.437944 14620 mvcc.cc:205] Tried to move back new op lower bound from 7298673264367345664 to 7298673264266018816. Current Snapshot: MvccSnapshot[applied={T|T < 7298673264367345664 or (T in {7298673264367345664})}]
I20260619 20:57:32.438180 14622 mvcc.cc:205] Tried to move back new op lower bound from 7298673264367345664 to 7298673264266018816. Current Snapshot: MvccSnapshot[applied={T|T < 7298673264367345664}]
I20260619 20:57:32.439828 14129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:32.439926 14129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:32.440692 14129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Sending DeleteTablet for 3 replicas of tablet 6329a3d3cd0b4afc9f435c9f74eb6d32
I20260619 20:57:32.441053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14192
I20260619 20:57:32.441038 14389 tablet_service.cc:1514] Processing DeleteTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49562
I20260619 20:57:32.441038 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38752
I20260619 20:57:32.445559 14630 tablet_replica.cc:330] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: stopping tablet replica
I20260619 20:57:32.445663 14630 raft_consensus.cc:2229] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 20:57:32.445716 14116 connection.cc:535] client connection to 127.0.98.1:40463 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:57:32.445757 14629 tablet_replica.cc:330] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: stopping tablet replica
I20260619 20:57:32.445770 14630 raft_consensus.cc:2258] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:57:32.445794 14116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:57:32.445838 14629 raft_consensus.cc:2229] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:32.445926 14629 raft_consensus.cc:2258] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:32.446077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14326
I20260619 20:57:32.446975 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.446811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40463 (local address 127.0.0.1:57726)
0619 20:57:32.446885 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.446942 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:57:32.447085 14629 ts_tablet_manager.cc:1905] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:57:32.447144 14116 catalog_manager.cc:4584] TS b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40463: connect: Connection refused (error 111)
I20260619 20:57:32.448622 14629 ts_tablet_manager.cc:1918] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:32.448736 14629 log.cc:1196] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/6329a3d3cd0b4afc9f435c9f74eb6d32
I20260619 20:57:32.448999 14629 ts_tablet_manager.cc:1939] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b2d987308c4341da87f4f7d44e35fad8: Deleting consensus metadata
I20260619 20:57:32.449373 14114 catalog_manager.cc:4841] TS b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (table test-table [id=1167b2d2a0e94c158909cadbfd9575b6]) successfully deleted
I20260619 20:57:32.451733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14460
I20260619 20:57:32.451864 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.451713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:49576)
0619 20:57:32.451799 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.451831 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:57:32.455833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14098
I20260619 20:57:32.460652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--webserver_interface=127.0.98.62
--webserver_port=39605
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:32.542508 14632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.542706 14632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.542725 14632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.543846 14632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:32.543884 14632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.543915 14632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.543926 14632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:32.543977 14632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:32.545408 14632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.545655 14632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.545874 14632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:32.554111 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.554073 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.554103 14632 server_base.cc:1034] running on GCE node
W20260619 20:57:32.554072 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.554847 14632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.555162 14632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.556324 14632 hybrid_clock.cc:648] HybridClock initialized: now 1781902652556320 us; error 37 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.558751 14632 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:32.559176 14632 webserver.cc:460] Webserver started at http://127.0.98.62:39605/ using document root <none> and password file <none>
I20260619 20:57:32.559378 14632 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.559438 14632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.560720 14632 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.561448 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.561633 14632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:32.561714 14632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "cf5d155aaeb8483388ac9d2f228abc39"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "c096dd89f5aa7a72c4cd10cdd139ae6c"
server_key_iv: "bc7b11f8e92cd3c382ff999dd969c19b"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.562040 14632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:32.606091 14632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:32.608528 14632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:32.608636 14632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:32.612380 14632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36083
I20260619 20:57:32.612432 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36083 every 8 connection(s)
I20260619 20:57:32.612787 14632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:32.614344 14701 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:57:32.614989 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap starting.
I20260619 20:57:32.615792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14632
I20260619 20:57:32.616066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40463
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=35051
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:32.623544 14701 log.cc:824] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.625344 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:32.625608 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap complete.
I20260619 20:57:32.627851 14701 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:32.628087 14701 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.628283 14701 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:32.628372 14701 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:32.628427 14701 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:32.628473 14701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:57:32.629467 14701 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:32.629573 14701 leader_election.cc:302] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5d155aaeb8483388ac9d2f228abc39; no voters:
I20260619 20:57:32.629774 14701 leader_election.cc:288] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:57:32.629805 14705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:57:32.629913 14705 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 LEADER]: Becoming Leader. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Running, Role: LEADER
I20260619 20:57:32.629992 14705 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:32.630023 14701 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:32.630280 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5d155aaeb8483388ac9d2f228abc39. Latest consensus state: current_term: 2 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:32.630369 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:32.631042 14709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:32.631489 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:32.631562 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:32.631814 14709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:32.631919 14709 catalog_manager.cc:1259] Loaded cluster ID: cc188e1f554d410b8d445560fd86cd09
I20260619 20:57:32.632162 14709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:57:32.632405 14709 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 20:57:32.637050 14709 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 20:57:32.639477 14709 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:32.639657 14709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Loaded TSK: 0
I20260619 20:57:32.639878 14709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:57:32.700575 14703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.700835 14703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.700874 14703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.701977 14703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.702037 14703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.702085 14703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:32.703598 14703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35051
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.703850 14703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.704070 14703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:32.709992 14728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.709942 14729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.710222 14731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.710242 14703 server_base.cc:1034] running on GCE node
I20260619 20:57:32.710462 14703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.710748 14703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.711932 14703 hybrid_clock.cc:648] HybridClock initialized: now 1781902652711918 us; error 26 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.714833 14703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:32.715245 14703 webserver.cc:460] Webserver started at http://127.0.98.1:35051/ using document root <none> and password file <none>
I20260619 20:57:32.715474 14703 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.715556 14703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.716886 14703 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.717582 14738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.717757 14703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.717845 14703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b4c9fed3d38242df9a36b548c8e7641c"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "f00fb8ed6070133af199d39a5d1c9d7a"
server_key_iv: "a9a1185a03d8ef4c74c7c16c3f74a87a"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.718170 14703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:32.744194 14703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:32.746541 14703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:32.746698 14703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:32.746934 14703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:32.747413 14745 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:57:32.748850 14703 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:57:32.748900 14703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:57:32.748935 14703 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:57:32.749572 14703 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:57:32.749615 14703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:57:32.749655 14745 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap starting.
I20260619 20:57:32.757337 14703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40463
I20260619 20:57:32.757498 14852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40463 every 8 connection(s)
I20260619 20:57:32.757849 14703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:32.760713 14745 log.cc:824] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.761134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14703
I20260619 20:57:32.761411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=35303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:32.763370 14745 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:32.763674 14745 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap complete.
I20260619 20:57:32.764814 14745 ts_tablet_manager.cc:1397] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260619 20:57:32.766009 14745 raft_consensus.cc:348] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.766268 14745 raft_consensus.cc:729] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c9fed3d38242df9a36b548c8e7641c, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.766428 14745 consensus_queue.cc:260] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.766686 14745 ts_tablet_manager.cc:1428] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:32.772543 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.758646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:54595)
0619 20:57:32.758966 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.759011 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.759576 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.760106 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.760110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.760496 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.760744 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.760749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.764226 (+ 3477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.764227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.765596 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.765599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.766600 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.766602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.766724 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.767345 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.767355 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.769068 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.770973 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.770979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.770981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.771201 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.771470 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.771471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.771471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.771573 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.771948 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.771953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.772084 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.772190 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.772355 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":98,"threads_started":1}
I20260619 20:57:32.772542 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.758706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54595 (local address 127.0.98.62:36083)
0619 20:57:32.758884 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.758893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.759159 (+ 266us) server_negotiation.cc:407] Connection header received
0619 20:57:32.759820 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.759827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.759898 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.760076 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.760871 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.764111 (+ 3240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.765699 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.766233 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.766488 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.769186 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.769189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.769190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.769200 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.770873 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.771297 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.771300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.771301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.771394 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.771655 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.771656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.771656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.771870 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.771954 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.772241 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.772361 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260619 20:57:32.773144 14853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:32.773228 14853 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:32.773404 14853 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:32.774078 14665 ts_manager.cc:194] Registered new tserver with Master: b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463)
I20260619 20:57:32.775205 14665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54595
I20260619 20:57:32.775997 14853 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:57:32.783103 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.774627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40463 (local address 127.0.0.1:57732)
0619 20:57:32.774833 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.774860 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.774915 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.775179 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.775186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.775503 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.775553 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.775556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.776959 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.776960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.777922 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.777924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.778390 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.778393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.778468 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.778900 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.778905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.780262 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.781990 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.781996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.781998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.782183 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.782430 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.782431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.782431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.782487 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.782767 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.782773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.782820 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.782907 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.783015 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260619 20:57:32.783267 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.774694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.98.1:40463)
0619 20:57:32.775023 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.775027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.775035 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:32.775064 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.775066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.775077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.775147 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.776131 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.776877 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.777996 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.778327 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.778380 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.780392 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.780399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.780399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.780411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.781891 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.782305 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.782307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.782308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.782364 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.782557 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.782558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.782558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.782689 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.782771 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.782953 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.783036 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":91,"threads_started":1}
I20260619 20:57:32.783589 14782 tablet_service.cc:1514] Processing DeleteTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57732
I20260619 20:57:32.784101 14862 tablet_replica.cc:330] stopping tablet replica
I20260619 20:57:32.784191 14862 raft_consensus.cc:2229] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:32.784258 14862 raft_consensus.cc:2258] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:32.784529 14862 ts_tablet_manager.cc:1905] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:32.786813 14862 ts_tablet_manager.cc:1918] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:32.786887 14862 log.cc:1196] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/6329a3d3cd0b4afc9f435c9f74eb6d32
I20260619 20:57:32.787091 14862 ts_tablet_manager.cc:1939] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P b4c9fed3d38242df9a36b548c8e7641c: Deleting consensus metadata
I20260619 20:57:32.787596 14651 catalog_manager.cc:4841] TS b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463): tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (table test-table [id=1167b2d2a0e94c158909cadbfd9575b6]) successfully deleted
W20260619 20:57:32.847589 14858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.847786 14858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.847810 14858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.848860 14858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.848892 14858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.848917 14858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:32.850257 14858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35303
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.850422 14858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.850579 14858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:32.855324 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.855333 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.855351 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.855566 14858 server_base.cc:1034] running on GCE node
I20260619 20:57:32.855762 14858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.855988 14858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.857149 14858 hybrid_clock.cc:648] HybridClock initialized: now 1781902652857141 us; error 48 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.859625 14858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:32.860097 14858 webserver.cc:460] Webserver started at http://127.0.98.2:35303/ using document root <none> and password file <none>
I20260619 20:57:32.860334 14858 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.860388 14858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.861812 14858 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.862491 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.862640 14858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:32.862728 14858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "14032d24e84541289b3e0008d3bb8bfe"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "aa95ae905b7be3ff876c8f28c1bb6778"
server_key_iv: "2b6e7b60e033a2765e20384134e03677"
server_key_version: "encryptionkey@0"
I20260619 20:57:32.863015 14858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:32.886812 14858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:32.888971 14858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:32.889075 14858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:32.889250 14858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:32.889660 14884 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:57:32.891103 14858 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:57:32.891264 14858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:57:32.891310 14858 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:57:32.891906 14858 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:57:32.891960 14858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:57:32.892009 14884 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Bootstrap starting.
I20260619 20:57:32.893416 14884 log.cc:824] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:32.896739 14884 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:32.897058 14884 tablet_bootstrap.cc:492] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Bootstrap complete.
I20260619 20:57:32.897226 14884 ts_tablet_manager.cc:1397] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 20:57:32.898763 14858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260619 20:57:32.898798 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260619 20:57:32.899129 14858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:57:32.899520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14858
I20260619 20:57:32.899385 14884 raft_consensus.cc:348] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.899755 14884 raft_consensus.cc:729] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14032d24e84541289b3e0008d3bb8bfe, State: Initialized, Role: FOLLOWER
I20260619 20:57:32.899804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43673
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=37189
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:32.899916 14884 consensus_queue.cc:260] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } }
I20260619 20:57:32.900161 14884 ts_tablet_manager.cc:1428] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:32.911187 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.901696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51087 (local address 127.0.98.62:36083)
0619 20:57:32.901760 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.901762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.901946 (+ 184us) server_negotiation.cc:407] Connection header received
0619 20:57:32.902137 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.902138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.902151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.902228 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.902839 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.903370 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.904986 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.905184 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.905235 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.908163 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.908165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.908166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.908174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.909790 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.910150 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.910151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.910152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.910252 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.910519 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.910520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.910520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.910668 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.910769 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.911003 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.911094 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:57:32.911223 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.901671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:51087)
0619 20:57:32.901833 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.901864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.902032 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.902284 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.902288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.902555 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.902742 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.902746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.903494 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.903495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.904896 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.904900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.905273 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.905274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.905426 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.905932 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.905943 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.908071 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.909868 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.909874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.909875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.910076 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.910327 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.910327 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.910328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.910418 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.910762 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.910767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.910875 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.910955 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.911066 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260619 20:57:32.911698 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:32.911808 14993 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:32.911988 14993 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:32.912541 14665 ts_manager.cc:194] Registered new tserver with Master: 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
I20260619 20:57:32.913039 14665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51087
I20260619 20:57:32.913779 14993 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 20:57:32.920359 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.912838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:49592)
0619 20:57:32.912883 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:32.912893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:32.912927 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:32.913258 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:32.913259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:32.913428 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:32.913469 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.913470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.914922 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.914922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.915462 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:32.915463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.916086 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.916088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:32.916136 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.916526 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:32.916531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:32.917625 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:32.919476 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.919477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.919477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.919636 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.919831 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:32.919831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:32.919831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:32.919879 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:32.920131 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:32.920132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:32.920164 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 20:57:32.920213 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:57:32.920300 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:57:32.920440 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:32.912913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.0.98.2:39351)
0619 20:57:32.913046 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:32.913050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:32.913058 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:32.913088 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:32.913089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:32.913102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:32.913537 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:32.913722 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.914849 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.915559 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:32.916017 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:32.916083 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:32.917712 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:32.917736 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:32.917737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:32.917743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:32.919399 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.919708 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.919711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.919711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.919769 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:32.919934 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:32.919934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:32.919935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:32.920068 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:32.920145 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:32.920253 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:57:32.920338 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260619 20:57:32.920764 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260619 20:57:32.921360 14999 tablet_replica.cc:330] stopping tablet replica
I20260619 20:57:32.921473 14999 raft_consensus.cc:2229] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:32.921563 14999 raft_consensus.cc:2258] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:32.921828 14999 ts_tablet_manager.cc:1905] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:32.923885 14999 ts_tablet_manager.cc:1918] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:32.923961 14999 log.cc:1196] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/6329a3d3cd0b4afc9f435c9f74eb6d32
I20260619 20:57:32.924176 14999 ts_tablet_manager.cc:1939] T 6329a3d3cd0b4afc9f435c9f74eb6d32 P 14032d24e84541289b3e0008d3bb8bfe: Deleting consensus metadata
I20260619 20:57:32.924602 14649 catalog_manager.cc:4841] TS 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): tablet 6329a3d3cd0b4afc9f435c9f74eb6d32 (table test-table [id=1167b2d2a0e94c158909cadbfd9575b6]) successfully deleted
W20260619 20:57:32.984115 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:32.984334 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:32.984372 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:32.985463 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:32.985528 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:32.985572 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:32.987010 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37189
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:32.987267 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:32.987488 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:32.992321 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.992328 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:32.992329 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:32.992623 14996 server_base.cc:1034] running on GCE node
I20260619 20:57:32.992799 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:32.993000 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:32.994171 14996 hybrid_clock.cc:648] HybridClock initialized: now 1781902652994160 us; error 24 us; skew 500 ppm
Jun 19 20:57:32 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:32.996851 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:32.997313 14996 webserver.cc:460] Webserver started at http://127.0.98.3:37189/ using document root <none> and password file <none>
I20260619 20:57:32.997578 14996 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:32.997668 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:32.999186 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:32.999763 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:32.999922 14996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:33.000025 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "b2d987308c4341da87f4f7d44e35fad8"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "01bb7fc49532859d9ea4c65b58ad6c2e"
server_key_iv: "00853cf8bb1e9d290848e8c0b4a69c51"
server_key_version: "encryptionkey@0"
I20260619 20:57:33.000417 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:33.023404 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:33.025133 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:33.025290 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:33.025502 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:33.025792 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:33.025846 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.025885 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:33.025914 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.031688 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43673
I20260619 20:57:33.031747 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43673 every 8 connection(s)
I20260619 20:57:33.032083 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:33.035120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 14996
Jun 19 20:57:33 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:33.042841 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.035503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.98.62:36083)
0619 20:57:33.035600 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.035602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.035615 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:57:33.035701 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.035702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.035711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.035768 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:33.036410 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.036881 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.037898 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.038098 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.038151 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.039396 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:33.039396 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:33.039397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:33.039403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:33.041495 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:33.041873 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:33.041874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:33.041875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:33.041950 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:33.042190 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:33.042191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:33.042191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:33.042448 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:33.042590 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.042677 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.042742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260619 20:57:33.043046 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53345 (local address 127.0.98.62:36083)
0619 20:57:33.032810 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.032812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.033015 (+ 203us) server_negotiation.cc:407] Connection header received
0619 20:57:33.033522 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.033523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.033537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.033590 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:33.034574 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.035077 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.036563 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.036779 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.036855 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.039329 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:33.039331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:33.039331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:33.039338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:33.041456 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:33.041968 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:33.041968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:33.041969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:33.042019 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:33.042306 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:33.042307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:33.042308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:33.042450 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:33.042522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.042879 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.042959 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:57:33.043222 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.032718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:53345)
0619 20:57:33.032904 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.032937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:33.033419 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:33.033655 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:33.033659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:33.034264 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:33.034469 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.034474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.035172 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.035174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.036472 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.036474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.036858 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.036860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.037000 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.037487 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:33.037496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:33.039234 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:33.041595 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:33.041602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:33.041605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:33.041898 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:33.042088 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:33.042089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:33.042090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:33.042249 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:33.042523 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:33.042528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:33.042715 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 20:57:33.042820 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:57:33.042986 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260619 20:57:33.043812 15128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36083
I20260619 20:57:33.043912 15128 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:33.044087 15128 heartbeater.cc:511] Master 127.0.98.62:36083 requested a full tablet report, sending...
I20260619 20:57:33.044427 14665 ts_manager.cc:194] Registered new tserver with Master: b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673)
I20260619 20:57:33.044780 14665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53345
I20260619 20:57:33.054436 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:57:33.056847 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.054767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.98.62:36083)
0619 20:57:33.054829 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.054831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.054911 (+ 80us) server_negotiation.cc:407] Connection header received
0619 20:57:33.054959 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.054960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.054987 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.055042 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:57:33.055148 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.055586 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.056250 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.056415 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.056460 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.056581 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:57:33.056661 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:57:33.056717 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.056747 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.056760 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:57:33.057135 14665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:57:33.058764 14665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:57:33.060714 14782 tablet_service.cc:1467] Processing CreateTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 (DEFAULT_TABLE table=test-table [id=4a644383c0b94ca3a9dd736725b778f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:33.060817 14927 tablet_service.cc:1467] Processing CreateTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 (DEFAULT_TABLE table=test-table [id=4a644383c0b94ca3a9dd736725b778f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:33.060904 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa34adc7daf416ca3f3dc43f0a00d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:33.060961 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa34adc7daf416ca3f3dc43f0a00d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:33.062618 14884 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Bootstrap starting.
I20260619 20:57:33.063030 14884 tablet_bootstrap.cc:654] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:33.063364 14745 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap starting.
I20260619 20:57:33.063674 14745 tablet_bootstrap.cc:654] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:33.064054 14884 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: No bootstrap required, opened a new log
I20260619 20:57:33.064119 14884 ts_tablet_manager.cc:1397] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:33.064231 14745 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: No bootstrap required, opened a new log
I20260619 20:57:33.064283 14745 ts_tablet_manager.cc:1397] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:33.064438 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.060515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43673 (local address 127.0.0.1:38762)
0619 20:57:33.060549 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.060560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:33.060595 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:33.060986 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:33.060987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:33.060989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:33.061031 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.061032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.062190 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.062191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.063214 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.063218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.064188 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.064189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.064265 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.064288 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 20:57:33.064324 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 20:57:33.064374 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 20:57:33.064424 14884 raft_consensus.cc:348] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.064528 14884 raft_consensus.cc:374] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:33.064579 14884 raft_consensus.cc:729] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14032d24e84541289b3e0008d3bb8bfe, State: Initialized, Role: FOLLOWER
I20260619 20:57:33.064635 14884 consensus_queue.cc:260] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.064631 14745 raft_consensus.cc:348] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.064729 14884 ts_tablet_manager.cc:1428] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:33.064702 14745 raft_consensus.cc:374] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:33.064764 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.060596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.0.98.3:43673)
0619 20:57:33.060786 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.060808 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.060818 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:57:33.060855 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.060857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.060872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.061105 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:33.061161 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.062084 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.063295 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.064030 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.064551 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.064620 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.064649 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.064688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260619 20:57:33.064754 14745 raft_consensus.cc:729] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c9fed3d38242df9a36b548c8e7641c, State: Initialized, Role: FOLLOWER
I20260619 20:57:33.064822 14745 consensus_queue.cc:260] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.064914 14745 ts_tablet_manager.cc:1428] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:33.065563 15062 tablet_service.cc:1467] Processing CreateTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 (DEFAULT_TABLE table=test-table [id=4a644383c0b94ca3a9dd736725b778f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:33.065920 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa34adc7daf416ca3f3dc43f0a00d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:33.068572 15137 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Bootstrap starting.
I20260619 20:57:33.069185 15137 tablet_bootstrap.cc:654] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:33.069689 15137 log.cc:824] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:33.070400 15137 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: No bootstrap required, opened a new log
I20260619 20:57:33.070480 15137 ts_tablet_manager.cc:1397] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:33.071776 15137 raft_consensus.cc:348] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.071934 15137 raft_consensus.cc:374] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:33.071979 15137 raft_consensus.cc:729] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d987308c4341da87f4f7d44e35fad8, State: Initialized, Role: FOLLOWER
I20260619 20:57:33.072099 15137 consensus_queue.cc:260] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.072402 15137 ts_tablet_manager.cc:1428] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:57:33.072403 15128 heartbeater.cc:503] Master 127.0.98.62:36083 was elected leader, sending a full tablet report...
W20260619 20:57:33.149680 14994 tablet.cc:2369] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:33.156591 15139 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:33.156704 15139 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.157048 15139 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
I20260619 20:57:33.160043 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.157227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.3:53357)
0619 20:57:33.157282 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.157295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:33.157344 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:33.157556 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:33.157557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:33.157559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:33.157641 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.157643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.158391 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.158392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.159167 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.159169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.159792 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.159867 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.159871 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:33.159926 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:57:33.159967 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:57:33.160038 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.157300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57779 (local address 127.0.98.1:40463)
0619 20:57:33.157348 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.157351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.157499 (+ 148us) server_negotiation.cc:407] Connection header received
0619 20:57:33.157578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.157579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.157607 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.157682 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:33.157861 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.158421 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159325 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.159708 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159794 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.159867 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.159931 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.159968 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:57:33.160069 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.157318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53357 (local address 127.0.98.2:39351)
0619 20:57:33.157360 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.157362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.157370 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:33.157404 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.157404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.157427 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.157503 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:33.157717 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.158322 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159261 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.159698 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159794 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.159867 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.159965 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.160006 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:57:33.160060 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.157229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40463 (local address 127.0.98.3:57779)
0619 20:57:33.157423 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.157434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:33.157494 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:33.157716 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:33.157717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:33.157719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:33.157788 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.157790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.158508 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.158508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.159245 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:33.159247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.159796 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.159797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:33.159861 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.159865 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:33.159924 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:57:33.159968 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":11,"thread_start_us":99,"threads_started":1}
I20260619 20:57:33.160388 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa34adc7daf416ca3f3dc43f0a00d92" candidate_uuid: "b2d987308c4341da87f4f7d44e35fad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c9fed3d38242df9a36b548c8e7641c" is_pre_election: true
I20260619 20:57:33.160392 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa34adc7daf416ca3f3dc43f0a00d92" candidate_uuid: "b2d987308c4341da87f4f7d44e35fad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14032d24e84541289b3e0008d3bb8bfe" is_pre_election: true
I20260619 20:57:33.160552 14802 raft_consensus.cc:2454] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d987308c4341da87f4f7d44e35fad8 in term 0.
I20260619 20:57:33.160565 14947 raft_consensus.cc:2454] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d987308c4341da87f4f7d44e35fad8 in term 0.
I20260619 20:57:33.160823 15017 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2d987308c4341da87f4f7d44e35fad8, b4c9fed3d38242df9a36b548c8e7641c; no voters:
I20260619 20:57:33.160991 15139 raft_consensus.cc:2781] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:57:33.161082 15139 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:57:33.161127 15139 raft_consensus.cc:3037] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:33.161867 15139 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.162009 15139 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [CANDIDATE]: Term 1 election: Requested vote from peers b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
I20260619 20:57:33.162178 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa34adc7daf416ca3f3dc43f0a00d92" candidate_uuid: "b2d987308c4341da87f4f7d44e35fad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c9fed3d38242df9a36b548c8e7641c"
I20260619 20:57:33.162186 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa34adc7daf416ca3f3dc43f0a00d92" candidate_uuid: "b2d987308c4341da87f4f7d44e35fad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14032d24e84541289b3e0008d3bb8bfe"
I20260619 20:57:33.162302 14802 raft_consensus.cc:3037] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:33.162307 14947 raft_consensus.cc:3037] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:33.162976 14947 raft_consensus.cc:2454] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d987308c4341da87f4f7d44e35fad8 in term 1.
I20260619 20:57:33.162987 14802 raft_consensus.cc:2454] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d987308c4341da87f4f7d44e35fad8 in term 1.
I20260619 20:57:33.163144 15015 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14032d24e84541289b3e0008d3bb8bfe, b2d987308c4341da87f4f7d44e35fad8; no voters:
I20260619 20:57:33.163236 15139 raft_consensus.cc:2781] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:33.163388 15139 raft_consensus.cc:686] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 1 LEADER]: Becoming Leader. State: Replica: b2d987308c4341da87f4f7d44e35fad8, State: Running, Role: LEADER
I20260619 20:57:33.163498 15139 consensus_queue.cc:237] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.164388 14665 catalog_manager.cc:5495] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b2d987308c4341da87f4f7d44e35fad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: UNKNOWN } } }
I20260619 20:57:33.179414 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.177364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.98.3:43673)
0619 20:57:33.177426 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:33.177428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:33.177435 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:57:33.177462 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:33.177463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:33.177478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:33.177534 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:57:33.177627 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.178078 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.178819 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:33.178997 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:33.179051 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:33.179164 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:57:33.179251 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:57:33.179304 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:33.179335 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:57:33.179350 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:57:33.181058 14947 raft_consensus.cc:1261] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Refusing update from remote peer b2d987308c4341da87f4f7d44e35fad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:33.181058 14802 raft_consensus.cc:1261] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Refusing update from remote peer b2d987308c4341da87f4f7d44e35fad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:33.181391 15141 consensus_queue.cc:1035] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:33.181515 15139 consensus_queue.cc:1035] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:33.183430 15144 mvcc.cc:205] Tried to move back new op lower bound from 7298673267427074048 to 7298673267358617600. Current Snapshot: MvccSnapshot[applied={T|T < 7298673267427074048}]
I20260619 20:57:33.184170 15146 mvcc.cc:205] Tried to move back new op lower bound from 7298673267427074048 to 7298673267358617600. Current Snapshot: MvccSnapshot[applied={T|T < 7298673267427074048}]
I20260619 20:57:33.185598 14664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:33.185674 14664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:33.186400 14664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Sending DeleteTablet for 3 replicas of tablet ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:33.186700 14782 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57732
I20260619 20:57:33.186719 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260619 20:57:33.186797 14862 tablet_replica.cc:330] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: stopping tablet replica
I20260619 20:57:33.186797 14999 tablet_replica.cc:330] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: stopping tablet replica
I20260619 20:57:33.186806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14703
I20260619 20:57:33.186837 14862 raft_consensus.cc:2229] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:33.186841 14999 raft_consensus.cc:2229] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:33.186879 14999 raft_consensus.cc:2258] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:33.186890 14862 raft_consensus.cc:2258] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:33.187688 15062 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa34adc7daf416ca3f3dc43f0a00d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38762
I20260619 20:57:33.187829 14999 ts_tablet_manager.cc:1905] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:33.187983 15154 tablet_replica.cc:330] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: stopping tablet replica
I20260619 20:57:33.188061 15154 raft_consensus.cc:2229] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:57:33.188133 15154 raft_consensus.cc:2258] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:33.188550 15154 ts_tablet_manager.cc:1905] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:33.189320 14999 ts_tablet_manager.cc:1918] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:33.189373 14999 log.cc:1196] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:33.189584 14999 ts_tablet_manager.cc:1939] T ffa34adc7daf416ca3f3dc43f0a00d92 P 14032d24e84541289b3e0008d3bb8bfe: Deleting consensus metadata
I20260619 20:57:33.189857 14649 catalog_manager.cc:4841] TS 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): tablet ffa34adc7daf416ca3f3dc43f0a00d92 (table test-table [id=4a644383c0b94ca3a9dd736725b778f3]) successfully deleted
I20260619 20:57:33.190124 15154 ts_tablet_manager.cc:1918] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:33.190169 15154 log.cc:1196] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:33.190490 15154 ts_tablet_manager.cc:1939] T ffa34adc7daf416ca3f3dc43f0a00d92 P b2d987308c4341da87f4f7d44e35fad8: Deleting consensus metadata
I20260619 20:57:33.190834 14649 catalog_manager.cc:4841] TS b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): tablet ffa34adc7daf416ca3f3dc43f0a00d92 (table test-table [id=4a644383c0b94ca3a9dd736725b778f3]) successfully deleted
W20260619 20:57:33.193943 14651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40463 (error 108)
I20260619 20:57:33.194353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14858
I20260619 20:57:33.194486 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.194343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40463 (local address 127.0.0.1:57746)
0619 20:57:33.194411 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.194458 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 20:57:33.195056 14651 catalog_manager.cc:4584] TS b4c9fed3d38242df9a36b548c8e7641c (127.0.98.1:40463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffa34adc7daf416ca3f3dc43f0a00d92: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40463: connect: Connection refused (error 111)
I20260619 20:57:33.199585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14996
I20260619 20:57:33.204778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 14632
I20260619 20:57:33.209705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--webserver_interface=127.0.98.62
--webserver_port=39605
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:33.288235 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:33.288439 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:33.288461 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:33.289515 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:33.289547 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:33.289561 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:33.289574 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:33.289587 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:33.290954 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:33.291138 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:33.291296 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:33.295893 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.295890 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.295893 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.295959 15156 server_base.cc:1034] running on GCE node
I20260619 20:57:33.296281 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:33.296625 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:33.297762 15156 hybrid_clock.cc:648] HybridClock initialized: now 1781902653297740 us; error 39 us; skew 500 ppm
Jun 19 20:57:33 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:33.300369 15156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:33.300803 15156 webserver.cc:460] Webserver started at http://127.0.98.62:39605/ using document root <none> and password file <none>
I20260619 20:57:33.301062 15156 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:33.301134 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:33.302441 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:57:33.303067 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.303254 15156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.303351 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "cf5d155aaeb8483388ac9d2f228abc39"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "c096dd89f5aa7a72c4cd10cdd139ae6c"
server_key_iv: "bc7b11f8e92cd3c382ff999dd969c19b"
server_key_version: "encryptionkey@0"
I20260619 20:57:33.303683 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:33.313223 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:33.314957 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:33.315117 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:33.319062 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36083
I20260619 20:57:33.319110 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36083 every 8 connection(s)
I20260619 20:57:33.319479 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:33.320890 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:57:33.321446 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap starting.
I20260619 20:57:33.323968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15156
I20260619 20:57:33.324283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40463
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=35051
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:33.329927 15224 log.cc:824] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:33.331961 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:33.332304 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39: Bootstrap complete.
I20260619 20:57:33.334481 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:33.334684 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Initialized, Role: FOLLOWER
I20260619 20:57:33.334841 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:33.334897 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:33.334967 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:33.335009 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:57:33.335927 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:33.336006 15224 leader_election.cc:302] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5d155aaeb8483388ac9d2f228abc39; no voters:
I20260619 20:57:33.336179 15224 leader_election.cc:288] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:57:33.336211 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:57:33.336316 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [term 3 LEADER]: Becoming Leader. State: Replica: cf5d155aaeb8483388ac9d2f228abc39, State: Running, Role: LEADER
I20260619 20:57:33.336396 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260619 20:57:33.336436 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:33.336643 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5d155aaeb8483388ac9d2f228abc39. Latest consensus state: current_term: 3 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:33.336725 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:33.336647 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cf5d155aaeb8483388ac9d2f228abc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5d155aaeb8483388ac9d2f228abc39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260619 20:57:33.336884 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5d155aaeb8483388ac9d2f228abc39 [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:33.336918 15232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:33.337706 15232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:33.337812 15232 catalog_manager.cc:1259] Loaded cluster ID: cc188e1f554d410b8d445560fd86cd09
I20260619 20:57:33.337848 15232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 20:57:33.338233 15232 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f07dc96e9ad google::LogMessage::Fail() at ??:0
@ 0x7f07dc97188a google::LogMessage::SendToLog() at ??:0
@ 0x7f07dc96e4de google::LogMessage::Flush() at ??:0
@ 0x7f07dc96fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f07e488f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f07e48988b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f07de83c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f07de831954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f07df6956db start_thread at ??:0
@ 0x7f07db46c61f clone at ??:0
W20260619 20:57:33.433820 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:33.434084 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:33.434125 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:33.435761 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:33.435818 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:33.435858 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:33.438022 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35051
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:33.438279 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:33.438524 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:33.443477 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.443486 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.443686 15226 server_base.cc:1034] running on GCE node
W20260619 20:57:33.443790 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.443991 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:33.444253 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:33.445415 15226 hybrid_clock.cc:648] HybridClock initialized: now 1781902653445396 us; error 26 us; skew 500 ppm
Jun 19 20:57:33 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:33.447849 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:33.448271 15226 webserver.cc:460] Webserver started at http://127.0.98.1:35051/ using document root <none> and password file <none>
I20260619 20:57:33.448480 15226 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:33.448547 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:33.449867 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:33.450613 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.450802 15226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:57:33.450902 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "b4c9fed3d38242df9a36b548c8e7641c"
format_stamp: "Formatted at 2026-06-19 20:57:31 on dist-test-slave-8bgz"
server_key: "f00fb8ed6070133af199d39a5d1c9d7a"
server_key_iv: "a9a1185a03d8ef4c74c7c16c3f74a87a"
server_key_version: "encryptionkey@0"
I20260619 20:57:33.451224 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:33.463332 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:33.465345 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:33.465494 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:33.465716 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:33.466117 15277 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:57:33.468075 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:57:33.468132 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:57:33.468190 15226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:57:33.468787 15226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:57:33.468840 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:57:33.468914 15277 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap starting.
I20260619 20:57:33.476432 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40463
I20260619 20:57:33.476567 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40463 every 8 connection(s)
I20260619 20:57:33.476838 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:33.479743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15226
I20260619 20:57:33.480012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=35303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:57:33.482643 15277 log.cc:824] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:33.485344 15277 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:57:33.485666 15277 tablet_bootstrap.cc:492] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Bootstrap complete.
I20260619 20:57:33.486901 15277 ts_tablet_manager.cc:1397] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20260619 20:57:33.488277 15277 raft_consensus.cc:348] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.488597 15277 raft_consensus.cc:729] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c9fed3d38242df9a36b548c8e7641c, State: Initialized, Role: FOLLOWER
I20260619 20:57:33.488761 15277 consensus_queue.cc:260] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:33.488998 15277 ts_tablet_manager.cc:1428] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260619 20:57:33.560034 15387 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:57:33.477585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:54893)
0619 20:57:33.477750 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.477790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:33.478196 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:33.559827 (+ 81631us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260619 20:57:33.560261 15385 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36083: Client connection negotiation failed: client connection to 127.0.98.62:36083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:57:33.560585 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.560470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:54577)
0619 20:57:33.560523 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.560565 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:57:33.560947 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.560866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:42223)
0619 20:57:33.560912 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.560934 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:57:33.561074 15385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:57:33.564541 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:33.564745 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:33.564793 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:33.565927 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:33.565990 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:33.566052 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:33.567565 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35303
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:33.567822 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:33.568059 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:33.572753 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.572731 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.572777 15388 server_base.cc:1034] running on GCE node
W20260619 20:57:33.572731 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.573101 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:33.573284 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:33.574396 15388 hybrid_clock.cc:648] HybridClock initialized: now 1781902653574389 us; error 26 us; skew 500 ppm
Jun 19 20:57:33 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:33.576627 15388 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:33.577095 15388 webserver.cc:460] Webserver started at http://127.0.98.2:35303/ using document root <none> and password file <none>
I20260619 20:57:33.577304 15388 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:33.577366 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:33.578614 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:57:33.579116 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.579272 15388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:57:33.579357 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "14032d24e84541289b3e0008d3bb8bfe"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "aa95ae905b7be3ff876c8f28c1bb6778"
server_key_iv: "2b6e7b60e033a2765e20384134e03677"
server_key_version: "encryptionkey@0"
I20260619 20:57:33.579665 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:33.603957 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:33.605829 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:33.605991 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:33.606262 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:33.606566 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:33.606621 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.606662 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:33.606688 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.612396 15388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260619 20:57:33.612454 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260619 20:57:33.612776 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:57:33.613762 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.613405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:36113)
0619 20:57:33.613550 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.613628 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20260619 20:57:33.614171 15519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36083: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
I20260619 20:57:33.614439 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.614354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:58695)
0619 20:57:33.614398 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.614423 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:57:33.614714 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.614638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:40971)
0619 20:57:33.614679 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.614701 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:57:33.614818 15519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:57:33.615124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15388
I20260619 20:57:33.615358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43673
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36083
--webserver_port=37189
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:33.693645 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:33.693833 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:33.693856 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:33.695021 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:33.695062 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:33.695116 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:33.696506 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37189
--tserver_master_addrs=127.0.98.62:36083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:33.696743 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:33.696955 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:33.701870 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.701879 15530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:33.701879 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:33.702301 15522 server_base.cc:1034] running on GCE node
I20260619 20:57:33.702526 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:33.702754 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:33.703923 15522 hybrid_clock.cc:648] HybridClock initialized: now 1781902653703905 us; error 37 us; skew 500 ppm
Jun 19 20:57:33 dist-test-slave-8bgz krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:33.706377 15522 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:33.706866 15522 webserver.cc:460] Webserver started at http://127.0.98.3:37189/ using document root <none> and password file <none>
I20260619 20:57:33.707109 15522 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:33.707178 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:33.708400 15522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:33.708966 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.709136 15522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.709230 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "b2d987308c4341da87f4f7d44e35fad8"
format_stamp: "Formatted at 2026-06-19 20:57:32 on dist-test-slave-8bgz"
server_key: "01bb7fc49532859d9ea4c65b58ad6c2e"
server_key_iv: "00853cf8bb1e9d290848e8c0b4a69c51"
server_key_version: "encryptionkey@0"
I20260619 20:57:33.709558 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:33.730962 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:33.732722 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:33.732877 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:33.733139 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:33.733420 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:33.733474 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.733520 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:33.733547 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:33.739194 15522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43673
I20260619 20:57:33.739238 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43673 every 8 connection(s)
I20260619 20:57:33.739603 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:33.739795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15522
I20260619 20:57:33.740828 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.740476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:37291)
0619 20:57:33.740634 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.740698 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260619 20:57:33.741230 15651 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36083: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
I20260619 20:57:33.741500 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.741404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:33647)
0619 20:57:33.741451 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.741476 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:57:33.741776 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:33.741693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:35411)
0619 20:57:33.741735 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:33.741757 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:57:33.741879 15651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:57:34.561618 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:34.561403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:56671)
0619 20:57:34.561519 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:34.561557 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260619 20:57:34.615270 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:34.615076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:44275)
0619 20:57:34.615201 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:34.615231 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260619 20:57:34.742422 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:34.742173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:48101)
0619 20:57:34.742351 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:34.742378 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":130,"threads_started":1}
I20260619 20:57:35.234690 15658 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:35.234808 15658 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:35.235176 15658 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
Jun 19 20:57:35 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 20:57:35 dist-test-slave-8bgz krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 20:57:35.245028 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.235362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43673 (local address 127.0.98.1:44437)
0619 20:57:35.235471 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:35.235480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:35.235561 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:35.236430 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:35.236433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:35.236863 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:35.237029 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:35.237035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.238093 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.238094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:35.238954 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:35.238956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.239584 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:35.239699 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:35.240323 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:35.240328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:35.241799 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:35.243612 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:35.243618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:35.243620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:35.243830 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:35.244060 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:35.244061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:35.244061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:35.244185 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:35.244525 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:35.244532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:35.244690 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 20:57:35.244810 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 20:57:35.244961 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":54,"thread_start_us":59,"threads_started":1}
I20260619 20:57:35.245128 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.235476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44437 (local address 127.0.98.3:43673)
0619 20:57:35.235679 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:35.235691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:35.235731 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:57:35.236174 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:35.236176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:35.236223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:35.236388 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:35.237139 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.237983 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239066 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.239466 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239607 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:35.241897 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:35.241901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:35.241902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:35.241914 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:35.243543 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:35.243900 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:35.243903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:35.243903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:35.243982 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:35.244256 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:35.244257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:35.244257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:35.244457 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:35.244538 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:35.244883 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 20:57:35.245011 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
W20260619 20:57:35.245862 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:35.246129 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.235520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.1:56065)
0619 20:57:35.235601 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:35.235610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:35.235660 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:35.236446 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:35.236446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:35.236870 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:35.237030 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:35.237035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.238083 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.238084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:35.238953 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:35.238955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.239569 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:35.239698 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:35.240270 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:35.240278 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:35.242306 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:35.244828 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:35.244829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:35.244829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:35.245080 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:35.245362 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:35.245363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:35.245363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:35.245426 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:35.245840 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:35.245841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:35.245885 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:57:35.245958 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:57:35.246043 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":59,"thread_start_us":34,"threads_started":1}
I20260619 20:57:35.246308 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.235632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56065 (local address 127.0.98.2:39351)
0619 20:57:35.235772 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:35.235787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:35.235804 (+ 17us) server_negotiation.cc:407] Connection header received
0619 20:57:35.236192 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:35.236194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:35.236227 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:35.236331 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:35.237150 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.237983 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239052 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:35.239467 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:35.239603 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:35.242434 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:35.242438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:35.242438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:35.242450 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:35.244753 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:35.245177 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:35.245180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:35.245180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:35.245288 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:35.245496 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:35.245497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:35.245497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:35.245787 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:35.245848 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:35.246047 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 20:57:35.246141 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
W20260619 20:57:35.246871 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:35.246917 15271 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c9fed3d38242df9a36b548c8e7641c; no voters: 14032d24e84541289b3e0008d3bb8bfe, b2d987308c4341da87f4f7d44e35fad8
I20260619 20:57:35.247023 15658 raft_consensus.cc:2726] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:57:35.562294 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.562116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:47191)
0619 20:57:35.562169 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:35.562248 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:57:35.615890 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.615699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:41537)
0619 20:57:35.615815 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:35.615851 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260619 20:57:35.743014 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:35.742823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:50061)
0619 20:57:35.742935 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:35.742967 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260619 20:57:36.562935 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:36.562746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:42651)
0619 20:57:36.562859 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:36.562890 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 20:57:36.616536 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:36.616307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:42193)
0619 20:57:36.616439 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:36.616493 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260619 20:57:36.743600 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:36.743411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:48459)
0619 20:57:36.743537 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:36.743567 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260619 20:57:36.815212 15668 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:36.815328 15668 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:36.815507 15668 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
W20260619 20:57:36.815979 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
W20260619 20:57:36.816064 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:36.816116 15271 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c9fed3d38242df9a36b548c8e7641c; no voters: 14032d24e84541289b3e0008d3bb8bfe, b2d987308c4341da87f4f7d44e35fad8
I20260619 20:57:36.816206 15668 raft_consensus.cc:2726] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:57:37.563632 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:37.563416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:54539)
0619 20:57:37.563551 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:37.563587 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":101,"threads_started":1}
I20260619 20:57:37.617143 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:37.616922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:52709)
0619 20:57:37.617065 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:37.617095 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260619 20:57:37.744150 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:37.743970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:53631)
0619 20:57:37.744088 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:37.744116 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260619 20:57:38.564368 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:38.564084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:45947)
0619 20:57:38.564256 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:38.564289 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260619 20:57:38.617758 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:38.617580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:46957)
0619 20:57:38.617692 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:38.617724 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260619 20:57:38.744879 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:38.744632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:51885)
0619 20:57:38.744807 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:38.744842 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260619 20:57:39.565024 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:39.564796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:47149)
0619 20:57:39.564928 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:39.564961 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 20:57:39.618587 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:39.618369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:54211)
0619 20:57:39.618472 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:39.618539 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260619 20:57:39.745478 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:39.745268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:53481)
0619 20:57:39.745409 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:39.745444 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260619 20:57:39.963387 15678 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:39.963486 15678 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:39.963619 15678 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
W20260619 20:57:39.964015 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
W20260619 20:57:39.964102 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:39.964150 15271 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c9fed3d38242df9a36b548c8e7641c; no voters: 14032d24e84541289b3e0008d3bb8bfe, b2d987308c4341da87f4f7d44e35fad8
I20260619 20:57:39.964231 15678 raft_consensus.cc:2726] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:57:40.565748 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:40.565525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:52059)
0619 20:57:40.565661 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:40.565700 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260619 20:57:40.619191 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:40.618983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:54765)
0619 20:57:40.619121 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:40.619154 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260619 20:57:40.746099 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:40.745893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:33921)
0619 20:57:40.746028 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:40.746061 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260619 20:57:41.566502 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:41.566304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:47175)
0619 20:57:41.566418 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:41.566455 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260619 20:57:41.619799 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:41.619600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:42447)
0619 20:57:41.619727 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:41.619761 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260619 20:57:41.746686 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:41.746481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:40833)
0619 20:57:41.746604 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:41.746637 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260619 20:57:42.567238 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:42.567003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:49077)
0619 20:57:42.567140 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:42.567194 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260619 20:57:42.620450 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:42.620252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:54353)
0619 20:57:42.620375 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:42.620410 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260619 20:57:42.747295 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:42.747106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:34847)
0619 20:57:42.747218 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:42.747249 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260619 20:57:43.568040 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:43.567836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:56733)
0619 20:57:43.567964 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:43.567996 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260619 20:57:43.621011 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:43.620832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:57831)
0619 20:57:43.620944 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:43.620975 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260619 20:57:43.748116 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:43.747914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:48439)
0619 20:57:43.748046 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:43.748080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260619 20:57:44.333047 15691 raft_consensus.cc:482] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:44.333159 15691 raft_consensus.cc:504] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d987308c4341da87f4f7d44e35fad8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43673 } } peers { permanent_uuid: "b4c9fed3d38242df9a36b548c8e7641c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40463 } } peers { permanent_uuid: "14032d24e84541289b3e0008d3bb8bfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260619 20:57:44.333356 15691 leader_election.cc:288] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673), 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351)
W20260619 20:57:44.333781 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14032d24e84541289b3e0008d3bb8bfe (127.0.98.2:39351): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
W20260619 20:57:44.333833 15271 leader_election.cc:341] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2d987308c4341da87f4f7d44e35fad8 (127.0.98.3:43673): Not found: Tablet not found: ffa34adc7daf416ca3f3dc43f0a00d92
I20260619 20:57:44.333849 15271 leader_election.cc:302] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c9fed3d38242df9a36b548c8e7641c; no voters: 14032d24e84541289b3e0008d3bb8bfe, b2d987308c4341da87f4f7d44e35fad8
I20260619 20:57:44.333925 15691 raft_consensus.cc:2726] T ffa34adc7daf416ca3f3dc43f0a00d92 P b4c9fed3d38242df9a36b548c8e7641c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:57:44.568791 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:44.568562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:35133)
0619 20:57:44.568712 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:44.568745 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260619 20:57:44.621778 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:44.621437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:43805)
0619 20:57:44.621675 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:44.621725 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":70,"threads_started":1}
I20260619 20:57:44.748736 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:44.748530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:55435)
0619 20:57:44.748652 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:44.748695 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260619 20:57:45.569510 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:45.569259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:39249)
0619 20:57:45.569419 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:45.569460 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
I20260619 20:57:45.622520 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:45.622347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:34723)
0619 20:57:45.622452 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:45.622485 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260619 20:57:45.749359 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:45.749151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:41759)
0619 20:57:45.749285 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:45.749318 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260619 20:57:46.570154 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:46.569900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:58941)
0619 20:57:46.570073 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:46.570108 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
I20260619 20:57:46.623117 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:46.622912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:51501)
0619 20:57:46.623041 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:46.623072 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260619 20:57:46.750092 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:46.749873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:58291)
0619 20:57:46.750013 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:46.750048 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260619 20:57:47.570832 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:47.570633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:36983)
0619 20:57:47.570759 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:47.570787 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260619 20:57:47.623759 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:47.623535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:38599)
0619 20:57:47.623684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:47.623718 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260619 20:57:47.750774 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:47.750591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:54773)
0619 20:57:47.750701 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:47.750734 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260619 20:57:48.571556 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:48.571329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.1:48437)
0619 20:57:48.571471 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:48.571509 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
I20260619 20:57:48.624428 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:48.624234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.2:57471)
0619 20:57:48.624358 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:48.624390 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260619 20:57:48.745784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15226
I20260619 20:57:48.750538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15388
I20260619 20:57:48.752019 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:48.751540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36083 (local address 127.0.98.3:55847)
0619 20:57:48.751931 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:48.751968 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":72,"threads_started":1}
I20260619 20:57:48.754730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15522
2026-06-19T20:57:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18220 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:57:48 dist-test-slave-8bgz krb5kdc[15709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:57:48 dist-test-slave-8bgz krb5kdc[15709](info): set up 2 sockets
Jun 19 20:57:48 dist-test-slave-8bgz krb5kdc[15709](info): commencing operation
krb5kdc: starting...
W20260619 20:57:49.798295 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:57:49 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:57:49.908560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:49.987387 15722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:49.987610 15722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:49.987646 15722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:49.988771 15722 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:57:49.988835 15722 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:57:49.988854 15722 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:57:49.988871 15722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:57:49.988890 15722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:49.988912 15722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:49.988945 15722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:57:49.988974 15722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:57:49.990446 15722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:49.990720 15722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:49.990940 15722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:57:49.995762 15722 server_base.cc:1034] running on GCE node
W20260619 20:57:49.995896 15730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:49.995898 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:49.995896 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:49.996281 15722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:49.996613 15722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:49.997821 15722 hybrid_clock.cc:648] HybridClock initialized: now 1781902669997804 us; error 45 us; skew 500 ppm
Jun 19 20:57:49 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:50.000684 15722 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:57:50.001329 15722 webserver.cc:460] Webserver started at http://127.0.98.62:40797/ using document root <none> and password file <none>
I20260619 20:57:50.001576 15722 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:50.001646 15722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:50.001789 15722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:50.002950 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "119559b53ae54004b38754ebb7425fff"
format_stamp: "Formatted at 2026-06-19 20:57:49 on dist-test-slave-8bgz"
server_key: "8dd800ad3ab805c18d63b13e97ac9e36"
server_key_iv: "372d1c780c8c69ac921651e0ae03c697"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.003331 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "119559b53ae54004b38754ebb7425fff"
format_stamp: "Formatted at 2026-06-19 20:57:49 on dist-test-slave-8bgz"
server_key: "8dd800ad3ab805c18d63b13e97ac9e36"
server_key_iv: "372d1c780c8c69ac921651e0ae03c697"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.004717 15722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:57:50.005501 15737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.005697 15722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:50.005805 15722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "119559b53ae54004b38754ebb7425fff"
format_stamp: "Formatted at 2026-06-19 20:57:49 on dist-test-slave-8bgz"
server_key: "8dd800ad3ab805c18d63b13e97ac9e36"
server_key_iv: "372d1c780c8c69ac921651e0ae03c697"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.005883 15722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:50.018834 15722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:50.020728 15722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:50.020900 15722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:50.025130 15722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40069
I20260619 20:57:50.025169 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40069 every 8 connection(s)
I20260619 20:57:50.025556 15722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:57:50.026482 15790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:50.028903 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap starting.
I20260619 20:57:50.029488 15790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:50.029732 15790 log.cc:824] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:50.030530 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: No bootstrap required, opened a new log
I20260619 20:57:50.032122 15790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:57:50.032263 15790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:50.032290 15790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Initialized, Role: FOLLOWER
I20260619 20:57:50.032395 15790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:57:50.032480 15790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:57:50.032521 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:57:50.032593 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:50.033156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15722
I20260619 20:57:50.033267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:57:50.033300 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:57:50.033418 15790 leader_election.cc:302] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119559b53ae54004b38754ebb7425fff; no voters:
I20260619 20:57:50.033499 392 external_mini_cluster.cc:1428] Setting key a7f22a8710922feba7499b14bd86b41c
I20260619 20:57:50.033599 15790 leader_election.cc:288] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:57:50.033653 15793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:50.033803 15793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 LEADER]: Becoming Leader. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Running, Role: LEADER
I20260619 20:57:50.033882 15790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:57:50.033939 15793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:57:50.034685 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:57:50.034730 15794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119559b53ae54004b38754ebb7425fff. Latest consensus state: current_term: 1 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:57:50.034852 15794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:50.035270 15804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:57:50.035543 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:57:50.035777 15804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:57:50.038375 15804 catalog_manager.cc:1347] Generated new cluster ID: c7ea86091a2448afaff6c92cc7165efb
I20260619 20:57:50.038425 15804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:50.046716 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.034249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.0.98.62:40069)
0619 20:57:50.034533 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.034543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.034582 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:57:50.035395 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.035403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.035468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.035734 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:50.036510 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.037841 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.038834 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.039446 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.039586 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.042423 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:50.042427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:50.042427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:50.042438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:50.044985 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.045427 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.045430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.045430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.045527 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.045750 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.045751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.045751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.046040 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:50.046154 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.046355 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.046501 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260619 20:57:50.059530 15804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:57:50.060077 15804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:57:50.067220 15804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Generated new TSK 0
I20260619 20:57:50.067407 15804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:57:50.094130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:50.172631 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:50.172813 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:50.172834 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:50.173954 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:50.173993 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:50.174017 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:57:50.175426 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:50.175642 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:50.175884 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:50.180596 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:50.180583 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:50.180599 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:50.180869 15816 server_base.cc:1034] running on GCE node
I20260619 20:57:50.181041 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:50.181250 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:50.182385 15816 hybrid_clock.cc:648] HybridClock initialized: now 1781902670182379 us; error 32 us; skew 500 ppm
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:50.184728 15816 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:57:50.185117 15816 webserver.cc:460] Webserver started at http://127.0.98.1:33839/ using document root <none> and password file <none>
I20260619 20:57:50.185317 15816 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:50.185379 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:50.185505 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:50.186537 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "244d19134f4b41ca90ca9bd48759615b"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "742f1dd7ccbe35513fd250f93bff4866"
server_key_iv: "1305f27304edffa58aaa0b6405853a2a"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.186865 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "244d19134f4b41ca90ca9bd48759615b"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "742f1dd7ccbe35513fd250f93bff4866"
server_key_iv: "1305f27304edffa58aaa0b6405853a2a"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.188107 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:57:50.188930 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.189136 15816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:57:50.189234 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "244d19134f4b41ca90ca9bd48759615b"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "742f1dd7ccbe35513fd250f93bff4866"
server_key_iv: "1305f27304edffa58aaa0b6405853a2a"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.189313 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:50.199543 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:50.201732 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:50.201879 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:50.202101 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:50.202404 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:50.202454 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.202484 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:50.202509 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.208074 15816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40771
I20260619 20:57:50.208138 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40771 every 8 connection(s)
I20260619 20:57:50.208424 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:57:50.208884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15816
I20260619 20:57:50.209018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:57:50.209216 392 external_mini_cluster.cc:1428] Setting key 5e0537fde6941f7b15f87ad311d5624c
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:57:50.219059 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.209815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54139 (local address 127.0.98.62:40069)
0619 20:57:50.209858 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.209860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.210131 (+ 271us) server_negotiation.cc:407] Connection header received
0619 20:57:50.210896 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.210897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.210909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.210958 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:50.212039 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.212842 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.213874 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.214231 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.214325 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.216315 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:50.216317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:50.216317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:50.216326 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:50.217852 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.218257 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.218259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.218259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.218329 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.218560 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.218560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.218561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.218664 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:50.218714 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.218937 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.218998 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:57:50.219238 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.209819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:54139)
0619 20:57:50.210003 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.210050 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.210782 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.211024 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.211028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.211775 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:50.211947 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.211952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.212926 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.212927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.213789 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.213792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.214332 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.214332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.214422 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.214942 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:50.214950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:50.216214 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:50.217951 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.217956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.217958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.218157 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.218387 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.218388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.218388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.218483 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.218722 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:50.218727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:50.218831 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.218909 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.219023 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260619 20:57:50.219810 15945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:57:50.219931 15945 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 20:57:50.220116 15945 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:57:50.220597 15754 ts_manager.cc:194] Registered new tserver with Master: 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771)
I20260619 20:57:50.221247 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:57:50.232810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:50.311439 15950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:50.311620 15950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:50.311642 15950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:50.312666 15950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:50.312698 15950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:50.312722 15950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:57:50.313990 15950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:50.314164 15950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:50.314320 15950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:50.318884 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:50.318912 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:50.318912 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:50.319106 15950 server_base.cc:1034] running on GCE node
I20260619 20:57:50.319362 15950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:50.319551 15950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:50.320680 15950 hybrid_clock.cc:648] HybridClock initialized: now 1781902670320676 us; error 31 us; skew 500 ppm
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:50.323098 15950 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:57:50.323539 15950 webserver.cc:460] Webserver started at http://127.0.98.2:37699/ using document root <none> and password file <none>
I20260619 20:57:50.323760 15950 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:50.323827 15950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:50.323984 15950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:50.324965 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "6fdad759ad5e44018a1768d17e5d29c2"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "e0db66b1840e4b82c91d8b0ef5ec2a30"
server_key_iv: "c321ef648e0be67f8293c00872fbdfbd"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.325312 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6fdad759ad5e44018a1768d17e5d29c2"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "e0db66b1840e4b82c91d8b0ef5ec2a30"
server_key_iv: "c321ef648e0be67f8293c00872fbdfbd"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.326632 15950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:57:50.327304 15965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.327464 15950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.327557 15950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "6fdad759ad5e44018a1768d17e5d29c2"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "e0db66b1840e4b82c91d8b0ef5ec2a30"
server_key_iv: "c321ef648e0be67f8293c00872fbdfbd"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.327636 15950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:50.346652 15950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:50.348304 15950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:50.348464 15950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:50.348667 15950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:50.348954 15950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:50.349007 15950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.349046 15950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:50.349073 15950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.356325 15950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33237
I20260619 20:57:50.356418 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33237 every 8 connection(s)
I20260619 20:57:50.356680 15950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:57:50.358134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 15950
I20260619 20:57:50.358269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:57:50.358454 392 external_mini_cluster.cc:1428] Setting key caf14c9bae2461a8e337a124dfc6001a
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:50.367070 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.357395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33755 (local address 127.0.98.62:40069)
0619 20:57:50.357453 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.357456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.357648 (+ 192us) server_negotiation.cc:407] Connection header received
0619 20:57:50.358183 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.358185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.358197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.358292 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:50.359243 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.360044 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.361417 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.361749 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.361819 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.364098 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:50.364100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:50.364100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:50.364108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:50.365748 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.366117 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.366119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.366119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.366178 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.366448 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.366449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.366450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.366593 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:50.366675 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.366902 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.366979 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:57:50.367102 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.357357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:33755)
0619 20:57:50.357519 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.357558 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.358058 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.358378 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.358382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.358990 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:50.359161 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.359165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.360130 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.360130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.361331 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.361334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.361828 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.361828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.361928 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.362443 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:50.362458 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:50.364007 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:50.365833 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.365838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.365840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.366042 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.366282 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.366283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.366283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.366379 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.366667 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:50.366671 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:50.366774 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.366851 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.366959 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:57:50.367601 16079 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:57:50.367679 16079 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:50.367805 16079 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:57:50.368093 15754 ts_manager.cc:194] Registered new tserver with Master: 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
I20260619 20:57:50.368548 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:57:50.382586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:57:50.465408 16084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:57:50.465610 16084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:57:50.465636 16084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:57:50.466835 16084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:57:50.466881 16084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:57:50.466908 16084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:57:50.468395 16084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:57:50.468644 16084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:57:50.468890 16084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:57:50.473984 16090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:57:50.474004 16089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:50.474107 16084 server_base.cc:1034] running on GCE node
W20260619 20:57:50.473984 16092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:57:50.474457 16084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:57:50.474723 16084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:57:50.475940 16084 hybrid_clock.cc:648] HybridClock initialized: now 1781902670475912 us; error 48 us; skew 500 ppm
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:57:50.478461 16084 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:57:50.478912 16084 webserver.cc:460] Webserver started at http://127.0.98.3:42499/ using document root <none> and password file <none>
I20260619 20:57:50.479102 16084 fs_manager.cc:359] Metadata directory not provided
I20260619 20:57:50.479140 16084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:57:50.479225 16084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:57:50.480250 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "479c6c344cc8455fb315a77e8784c82c"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "965022ee40118305350b2b1e0b93e898"
server_key_iv: "9ab31237aedfda811f83aa0b044dfa7f"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.480536 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "479c6c344cc8455fb315a77e8784c82c"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "965022ee40118305350b2b1e0b93e898"
server_key_iv: "9ab31237aedfda811f83aa0b044dfa7f"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.481710 16084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:57:50.482379 16099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.482527 16084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:57:50.482611 16084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "479c6c344cc8455fb315a77e8784c82c"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "965022ee40118305350b2b1e0b93e898"
server_key_iv: "9ab31237aedfda811f83aa0b044dfa7f"
server_key_version: "encryptionkey@0"
I20260619 20:57:50.482663 16084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:57:50.495572 16084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:57:50.497751 16084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:57:50.497862 16084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:57:50.498042 16084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:57:50.498404 16084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:57:50.498440 16084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.498464 16084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:57:50.498509 16084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:57:50.504014 16084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43241
I20260619 20:57:50.504088 16212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43241 every 8 connection(s)
I20260619 20:57:50.504381 16084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:57:50.507731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16084
I20260619 20:57:50.507835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:57:50.508031 392 external_mini_cluster.cc:1428] Setting key bc7a08c46a3ba92f1f21013421b9c2b2
Jun 19 20:57:50 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:57:50.513693 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.505155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47521 (local address 127.0.98.62:40069)
0619 20:57:50.505232 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.505234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.505424 (+ 190us) server_negotiation.cc:407] Connection header received
0619 20:57:50.505920 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.505921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.505932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.505984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:50.507026 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.507504 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.508758 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.508978 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.509064 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.511166 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:50.511168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:50.511168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:50.511175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:50.512325 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.512730 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.512731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.512731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.512791 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.513046 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.513047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.513047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.513163 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:50.513231 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.513525 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.513602 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:57:50.513885 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.505112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:47521)
0619 20:57:50.505281 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.505314 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.505818 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.506063 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.506067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.506755 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:57:50.506939 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.506943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.507600 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.507601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.508676 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.508678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.509072 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.509073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.509209 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.509652 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:57:50.509659 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:57:50.511075 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:57:50.512423 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.512429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.512431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.512654 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.512867 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:57:50.512867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:57:50.512868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:57:50.512972 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:57:50.513236 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:57:50.513242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:57:50.513375 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.513501 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.513655 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260619 20:57:50.514573 16213 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:57:50.514695 16213 heartbeater.cc:464] Registering TS with master...
I20260619 20:57:50.514895 16213 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:57:50.515307 15754 ts_manager.cc:194] Registered new tserver with Master: 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241)
I20260619 20:57:50.515695 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47521
I20260619 20:57:50.519690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:57:50.526572 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.520992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.98.62:40069)
0619 20:57:50.521029 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.521031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.521133 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:57:50.521196 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.521197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.521205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.521252 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:57:50.521821 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.522360 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.523052 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.523229 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.523282 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.524320 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:57:50.524321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:57:50.524321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:57:50.524327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:57:50.525676 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.525938 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.525939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.525939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.525982 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:57:50.526117 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:57:50.526118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:57:50.526118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:57:50.526295 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:57:50.526374 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.526461 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.526514 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:57:50.528151 15754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:57:50.529626 15754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:57:50.536057 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.531999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.98.3:43241)
0619 20:57:50.532179 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.532184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.532195 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:57:50.532232 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.532234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.532249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.532507 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:50.532574 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.533621 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.534946 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.535682 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.535760 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.535820 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.535947 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.535980 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260619 20:57:50.536065 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.531766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.0.1:33108)
0619 20:57:50.531945 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.531997 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.532062 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.532367 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.532370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.532373 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:50.532432 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.532434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.533740 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.533741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.534853 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.534855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.535763 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.535764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.535842 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.535855 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.535930 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.535987 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260619 20:57:50.536402 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.532133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40771 (local address 127.0.0.1:56248)
0619 20:57:50.532251 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.532259 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.532357 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.532787 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.532788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.532790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:50.532841 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.532843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.534126 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.534127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.535294 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.535298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.536192 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.536193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.536246 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.536250 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.536296 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.536337 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260619 20:57:50.536639 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.98.1:40771)
0619 20:57:50.532214 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.532219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.532332 (+ 113us) server_negotiation.cc:407] Connection header received
0619 20:57:50.532447 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.532450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.532474 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.532548 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:50.532923 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.534022 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.535383 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.536038 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.536098 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.536157 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.536515 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.536563 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":144,"threads_started":1}
I20260619 20:57:50.536998 16147 tablet_service.cc:1467] Processing CreateTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (DEFAULT_TABLE table=test-table [id=d363125592cc4a309d0ca4d490f07d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:50.537359 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:50.537314 15879 tablet_service.cc:1467] Processing CreateTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (DEFAULT_TABLE table=test-table [id=d363125592cc4a309d0ca4d490f07d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:50.537622 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:50.538764 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.531807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33237 (local address 127.0.0.1:36362)
0619 20:57:50.532236 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.532246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.532313 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.533383 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.533384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.533386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:50.533421 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.533422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.535051 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.535052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.536147 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.536150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.538515 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.538516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.538584 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.538588 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.538653 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.538690 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":53,"threads_started":1}
I20260619 20:57:50.538786 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.532595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.2:33237)
0619 20:57:50.533128 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.533132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.533142 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:57:50.533201 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.533204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.533218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.533337 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:50.533577 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.534971 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.536217 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.538423 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.538496 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.538548 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.538675 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.538713 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":137,"threads_started":1}
I20260619 20:57:50.539564 16013 tablet_service.cc:1467] Processing CreateTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (DEFAULT_TABLE table=test-table [id=d363125592cc4a309d0ca4d490f07d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:57:50.539865 16013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:57:50.540697 16227 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Bootstrap starting.
I20260619 20:57:50.541172 16227 tablet_bootstrap.cc:654] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:50.541612 16227 log.cc:824] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:50.541848 16228 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Bootstrap starting.
I20260619 20:57:50.542397 16227 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: No bootstrap required, opened a new log
I20260619 20:57:50.542472 16228 tablet_bootstrap.cc:654] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:50.542471 16227 ts_tablet_manager.cc:1397] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:50.543148 16228 log.cc:824] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:50.543563 16230 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Bootstrap starting.
I20260619 20:57:50.543843 16228 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: No bootstrap required, opened a new log
I20260619 20:57:50.543936 16228 ts_tablet_manager.cc:1397] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:57:50.544145 16230 tablet_bootstrap.cc:654] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Neither blocks nor log segments found. Creating new log.
I20260619 20:57:50.544687 16230 log.cc:824] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:57:50.544615 16227 raft_consensus.cc:348] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.544780 16227 raft_consensus.cc:374] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:50.544822 16227 raft_consensus.cc:729] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479c6c344cc8455fb315a77e8784c82c, State: Initialized, Role: FOLLOWER
I20260619 20:57:50.544996 16227 consensus_queue.cc:260] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.545286 16227 ts_tablet_manager.cc:1428] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:57:50.545394 16230 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: No bootstrap required, opened a new log
I20260619 20:57:50.545423 16213 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
I20260619 20:57:50.545459 16230 ts_tablet_manager.cc:1397] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:57:50.545604 16228 raft_consensus.cc:348] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.545784 16228 raft_consensus.cc:374] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:50.545825 16228 raft_consensus.cc:729] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d19134f4b41ca90ca9bd48759615b, State: Initialized, Role: FOLLOWER
I20260619 20:57:50.545928 16228 consensus_queue.cc:260] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.546187 16228 ts_tablet_manager.cc:1428] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:57:50.546322 15945 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
I20260619 20:57:50.547194 16230 raft_consensus.cc:348] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.547312 16230 raft_consensus.cc:374] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:57:50.547333 16230 raft_consensus.cc:729] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdad759ad5e44018a1768d17e5d29c2, State: Initialized, Role: FOLLOWER
I20260619 20:57:50.547410 16230 consensus_queue.cc:260] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.547639 16230 ts_tablet_manager.cc:1428] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:57:50.547729 16079 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
W20260619 20:57:50.607245 16080 tablet.cc:2369] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:57:50.633575 16234 raft_consensus.cc:482] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:57:50.633695 16234 raft_consensus.cc:504] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.634140 16234 leader_election.cc:288] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
I20260619 20:57:50.637022 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.634399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42665 (local address 127.0.98.2:33237)
0619 20:57:50.634441 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.634443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.634451 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:57:50.634543 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.634545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.634557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.634627 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:50.634793 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.635348 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636291 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.636666 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.636758 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.636914 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.636955 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:57:50.637022 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.634325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33237 (local address 127.0.98.1:42665)
0619 20:57:50.634398 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.634411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.634458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.634633 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.634634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.634635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:50.634711 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.634717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.635427 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.635428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.636196 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.636198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.636748 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.636839 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.636842 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.636909 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.636957 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 20:57:50.637288 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.634364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55317 (local address 127.0.98.3:43241)
0619 20:57:50.634409 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.634411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.634703 (+ 292us) server_negotiation.cc:407] Connection header received
0619 20:57:50.634790 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.634791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.634803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.634883 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:57:50.635072 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.635589 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636496 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.636874 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.636963 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.637167 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.637218 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:57:50.637333 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b538a6564134d1cbe1d48f1eb7ba2bd" candidate_uuid: "244d19134f4b41ca90ca9bd48759615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdad759ad5e44018a1768d17e5d29c2" is_pre_election: true
I20260619 20:57:50.637470 16033 raft_consensus.cc:2454] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d19134f4b41ca90ca9bd48759615b in term 0.
I20260619 20:57:50.637511 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.634327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.98.1:55317)
0619 20:57:50.634626 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.634635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:57:50.634712 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:57:50.634886 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:57:50.634887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:57:50.634888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:57:50.634996 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.634998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.635666 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.635667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.636405 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:57:50.636407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.636968 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.636969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:57:50.637045 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.637047 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:57:50.637398 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 20:57:50.637444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":134,"thread_start_us":99,"threads_started":1}
I20260619 20:57:50.637708 15834 leader_election.cc:302] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b, 6fdad759ad5e44018a1768d17e5d29c2; no voters:
I20260619 20:57:50.637830 16234 raft_consensus.cc:2781] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:57:50.637785 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b538a6564134d1cbe1d48f1eb7ba2bd" candidate_uuid: "244d19134f4b41ca90ca9bd48759615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c6c344cc8455fb315a77e8784c82c" is_pre_election: true
I20260619 20:57:50.637917 16234 raft_consensus.cc:482] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:57:50.637946 16234 raft_consensus.cc:3037] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:50.637988 16167 raft_consensus.cc:2454] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d19134f4b41ca90ca9bd48759615b in term 0.
I20260619 20:57:50.638784 16234 raft_consensus.cc:504] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.638979 16234 leader_election.cc:288] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 1 election: Requested vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
I20260619 20:57:50.639127 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b538a6564134d1cbe1d48f1eb7ba2bd" candidate_uuid: "244d19134f4b41ca90ca9bd48759615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c6c344cc8455fb315a77e8784c82c"
I20260619 20:57:50.639158 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b538a6564134d1cbe1d48f1eb7ba2bd" candidate_uuid: "244d19134f4b41ca90ca9bd48759615b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdad759ad5e44018a1768d17e5d29c2"
I20260619 20:57:50.639205 16167 raft_consensus.cc:3037] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:50.639245 16033 raft_consensus.cc:3037] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:57:50.639907 16033 raft_consensus.cc:2454] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d19134f4b41ca90ca9bd48759615b in term 1.
I20260619 20:57:50.639930 16167 raft_consensus.cc:2454] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d19134f4b41ca90ca9bd48759615b in term 1.
I20260619 20:57:50.640110 15833 leader_election.cc:302] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b, 479c6c344cc8455fb315a77e8784c82c; no voters:
I20260619 20:57:50.640200 16234 raft_consensus.cc:2781] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:57:50.640319 16234 raft_consensus.cc:686] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 LEADER]: Becoming Leader. State: Replica: 244d19134f4b41ca90ca9bd48759615b, State: Running, Role: LEADER
I20260619 20:57:50.640411 16234 consensus_queue.cc:237] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:57:50.641217 15753 catalog_manager.cc:5495] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d19134f4b41ca90ca9bd48759615b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "244d19134f4b41ca90ca9bd48759615b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } health_report { overall_health: UNKNOWN } } }
I20260619 20:57:50.649658 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.647691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.98.1:40771)
0619 20:57:50.647758 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:57:50.647760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:57:50.647766 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:57:50.647818 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:57:50.647818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:57:50.647832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:57:50.647881 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:57:50.647977 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.648384 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.649061 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:57:50.649227 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:57:50.649274 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:57:50.649389 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:57:50.649481 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:57:50.649532 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:57:50.649566 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 20:57:50.649579 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:57:50.651270 16033 raft_consensus.cc:1261] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Refusing update from remote peer 244d19134f4b41ca90ca9bd48759615b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:50.651270 16167 raft_consensus.cc:1261] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Refusing update from remote peer 244d19134f4b41ca90ca9bd48759615b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:57:50.651568 16234 consensus_queue.cc:1035] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:50.651698 16237 consensus_queue.cc:1035] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [LEADER]: Connected to new peer: Peer: permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:57:50.654232 16240 mvcc.cc:205] Tried to move back new op lower bound from 7298673338985340928 to 7298673338943729664. Current Snapshot: MvccSnapshot[applied={T|T < 7298673338985340928}]
I20260619 20:57:50.654325 16241 mvcc.cc:205] Tried to move back new op lower bound from 7298673338985340928 to 7298673338943729664. Current Snapshot: MvccSnapshot[applied={T|T < 7298673338985340928}]
I20260619 20:57:50.654724 16242 mvcc.cc:205] Tried to move back new op lower bound from 7298673338985340928 to 7298673338943729664. Current Snapshot: MvccSnapshot[applied={T|T < 7298673338985340928}]
I20260619 20:57:50.656250 15753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:50.656340 15753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:57:50.657052 15753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Sending DeleteTablet for 3 replicas of tablet 4b538a6564134d1cbe1d48f1eb7ba2bd
I20260619 20:57:50.657359 15879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56248
I20260619 20:57:50.657375 16013 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36362
I20260619 20:57:50.657691 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33108
I20260619 20:57:50.657707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15816
I20260619 20:57:50.659020 16249 tablet_replica.cc:330] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: stopping tablet replica
I20260619 20:57:50.659121 16249 raft_consensus.cc:2229] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:50.659217 16249 raft_consensus.cc:2258] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:50.659644 16249 ts_tablet_manager.cc:1905] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:57:50.661078 16249 ts_tablet_manager.cc:1918] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:50.661135 16249 log.cc:1196] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/4b538a6564134d1cbe1d48f1eb7ba2bd
I20260619 20:57:50.661386 16249 ts_tablet_manager.cc:1939] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 6fdad759ad5e44018a1768d17e5d29c2: Deleting consensus metadata
I20260619 20:57:50.661765 15740 catalog_manager.cc:4841] TS 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (table test-table [id=d363125592cc4a309d0ca4d490f07d54]) successfully deleted
I20260619 20:57:50.662320 16251 tablet_replica.cc:330] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: stopping tablet replica
I20260619 20:57:50.662412 16251 raft_consensus.cc:2229] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:57:50.662521 16251 raft_consensus.cc:2258] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:57:50.662878 16251 ts_tablet_manager.cc:1905] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:57:50.663421 15741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40771 (error 108)
I20260619 20:57:50.663911 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:57:50.663777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40771 (local address 127.0.0.1:56260)
0619 20:57:50.663845 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:57:50.663882 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:57:50.663967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15950
I20260619 20:57:50.664582 16251 ts_tablet_manager.cc:1918] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:57:50.664655 16251 log.cc:1196] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/4b538a6564134d1cbe1d48f1eb7ba2bd
I20260619 20:57:50.664916 16251 ts_tablet_manager.cc:1939] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 479c6c344cc8455fb315a77e8784c82c: Deleting consensus metadata
I20260619 20:57:50.665303 15739 catalog_manager.cc:4841] TS 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (table test-table [id=d363125592cc4a309d0ca4d490f07d54]) successfully deleted
W20260619 20:57:50.666700 15741 catalog_manager.cc:4584] TS 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40771: connect: Connection refused (error 111)
I20260619 20:57:50.669780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16084
I20260619 20:57:50.674474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 15722
I20260619 20:58:10.679569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--webserver_interface=127.0.98.62
--webserver_port=40797
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:10.758242 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:10.758435 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:10.758456 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:10.759547 16253 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:58:10.759581 16253 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:58:10.759594 16253 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:58:10.759605 16253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:10.759618 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:10.759631 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:10.759644 16253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:10.759660 16253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:10.761021 16253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:10.761199 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:10.761353 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:10.769552 16262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:10.769595 16259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:10.769595 16260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:10.769794 16253 server_base.cc:1034] running on GCE node
I20260619 20:58:10.770071 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:10.770450 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:10.771618 16253 hybrid_clock.cc:648] HybridClock initialized: now 1781902690771605 us; error 33 us; skew 500 ppm
Jun 19 20:58:10 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:10.774019 16253 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:58:10.774585 16253 webserver.cc:460] Webserver started at http://127.0.98.62:40797/ using document root <none> and password file <none>
I20260619 20:58:10.774776 16253 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:10.774827 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:10.776144 16253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:10.776731 16269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:10.776914 16253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:10.776986 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "119559b53ae54004b38754ebb7425fff"
format_stamp: "Formatted at 2026-06-19 20:57:49 on dist-test-slave-8bgz"
server_key: "8dd800ad3ab805c18d63b13e97ac9e36"
server_key_iv: "372d1c780c8c69ac921651e0ae03c697"
server_key_version: "encryptionkey@0"
I20260619 20:58:10.777369 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:10.799836 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:10.802062 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:10.802212 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:10.806262 16253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40069
I20260619 20:58:10.806315 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40069 every 8 connection(s)
I20260619 20:58:10.806691 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:10.808112 16322 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:58:10.808753 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap starting.
I20260619 20:58:10.814570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16253
I20260619 20:58:10.814845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40771
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=33839
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:58:10.816756 16322 log.cc:824] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:10.818622 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:58:10.818913 16322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap complete.
I20260619 20:58:10.821313 16322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:10.821561 16322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Initialized, Role: FOLLOWER
I20260619 20:58:10.821745 16322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:10.821821 16322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:10.821879 16322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:10.821928 16322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:58:10.822994 16322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:10.823114 16322 leader_election.cc:302] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119559b53ae54004b38754ebb7425fff; no voters:
I20260619 20:58:10.823323 16322 leader_election.cc:288] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:58:10.823359 16326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:58:10.823486 16326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 LEADER]: Becoming Leader. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Running, Role: LEADER
I20260619 20:58:10.823596 16322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:10.823632 16326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:10.823925 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119559b53ae54004b38754ebb7425fff. Latest consensus state: current_term: 2 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:58:10.823926 16328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:58:10.824465 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:10.824488 16328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:10.825320 16342 catalog_manager.cc:1259] Loaded cluster ID: c7ea86091a2448afaff6c92cc7165efb
I20260619 20:58:10.825380 16342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: loading cluster ID for follower catalog manager: success
I20260619 20:58:10.826251 16342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: acquiring CA information for follower catalog manager: success
I20260619 20:58:10.826611 16342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:58:10.826686 16337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:10.826992 16337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:10.827086 16337 catalog_manager.cc:1259] Loaded cluster ID: c7ea86091a2448afaff6c92cc7165efb
I20260619 20:58:10.827142 16337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:10.827319 16337 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:10.827417 16337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Loaded TSK: 0
I20260619 20:58:10.835569 16337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Generated new TSK 1
I20260619 20:58:10.835647 16337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:58:10.910127 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:10.910411 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:10.910450 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:10.911567 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:10.911634 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:10.911675 16324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:10.913131 16324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33839
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:10.913373 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:10.913592 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:10.918522 16350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:10.918661 16352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:10.918910 16324 server_base.cc:1034] running on GCE node
W20260619 20:58:10.918534 16349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:10.919173 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:10.919441 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:10.920598 16324 hybrid_clock.cc:648] HybridClock initialized: now 1781902690920588 us; error 27 us; skew 500 ppm
Jun 19 20:58:10 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:10.923125 16324 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:58:10.923635 16324 webserver.cc:460] Webserver started at http://127.0.98.1:33839/ using document root <none> and password file <none>
I20260619 20:58:10.923859 16324 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:10.923914 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:10.925489 16324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:10.926167 16359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:10.926365 16324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:10.926467 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "244d19134f4b41ca90ca9bd48759615b"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "742f1dd7ccbe35513fd250f93bff4866"
server_key_iv: "1305f27304edffa58aaa0b6405853a2a"
server_key_version: "encryptionkey@0"
I20260619 20:58:10.926797 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:10.958312 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:10.960774 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:10.960916 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:10.961189 16324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:10.961635 16366 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:58:10.963080 16324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:58:10.963126 16324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:58:10.963152 16324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:58:10.963784 16324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:58:10.963817 16324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:58:10.963917 16366 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Bootstrap starting.
I20260619 20:58:10.970363 16324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40771
I20260619 20:58:10.970465 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40771 every 8 connection(s)
I20260619 20:58:10.970777 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:10.974061 16366 log.cc:824] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:10.976477 16366 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:58:10.976765 16366 tablet_bootstrap.cc:492] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Bootstrap complete.
I20260619 20:58:10.977922 16366 ts_tablet_manager.cc:1397] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
Jun 19 20:58:10 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:10.979028 16366 raft_consensus.cc:348] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:58:10.979280 16366 raft_consensus.cc:729] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d19134f4b41ca90ca9bd48759615b, State: Initialized, Role: FOLLOWER
I20260619 20:58:10.979430 16366 consensus_queue.cc:260] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } } peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } }
I20260619 20:58:10.979703 16366 ts_tablet_manager.cc:1428] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260619 20:58:10.980067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16324
I20260619 20:58:10.980340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33237
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=37699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:58:10.982858 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:10.971552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47165 (local address 127.0.98.62:40069)
0619 20:58:10.971736 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:10.971745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:10.971962 (+ 217us) server_negotiation.cc:407] Connection header received
0619 20:58:10.972820 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:10.972825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:10.972866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:10.972982 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:10.973885 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.975072 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.976476 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.976929 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.977013 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:10.979286 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:10.979289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:10.979290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:10.979301 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:10.981198 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:10.981605 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:10.981608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:10.981609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:10.981706 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:10.981955 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:10.981956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:10.981956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:10.982165 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:10.982300 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:10.982574 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 20:58:10.982691 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260619 20:58:10.982857 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:10.971519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:47165)
0619 20:58:10.971771 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:10.971822 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:10.972635 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:10.973066 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:10.973070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:10.973515 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:10.973782 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:10.973788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.975186 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.975187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:10.976390 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:10.976392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.977029 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.977030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:10.977172 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:10.977684 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:10.977693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:10.979180 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:10.981286 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:10.981292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:10.981294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:10.981524 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:10.981776 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:10.981776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:10.981777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:10.981883 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:10.982279 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:10.982286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:10.982415 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 20:58:10.982523 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 20:58:10.982682 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260619 20:58:10.983434 16474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:58:10.983536 16474 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:10.983712 16474 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:58:10.984395 16286 ts_manager.cc:194] Registered new tserver with Master: 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771)
I20260619 20:58:10.985433 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47165
I20260619 20:58:10.986456 16474 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
Jun 19 20:58:10 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:58:10.993088 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:10.984977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40771 (local address 127.0.0.1:35026)
0619 20:58:10.985734 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:10.985765 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:10.985832 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:10.986129 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:10.986135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:10.986508 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:10.986559 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:10.986561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.987518 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.987519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:10.988071 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:10.988073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.988543 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.988546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:10.988628 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:10.989080 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:10.989084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:10.990365 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:10.991914 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:10.991920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:10.991922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:10.992141 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:10.992350 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:10.992351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:10.992351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:10.992408 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:10.992721 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:10.992727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:10.992776 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:58:10.992876 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 20:58:10.992992 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":75,"threads_started":1}
I20260619 20:58:10.993088 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:10.985041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.0.98.1:40771)
0619 20:58:10.985953 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:10.985957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:10.985967 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:10.985995 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:10.985997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:10.986009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:10.986069 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:10.986624 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.987423 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.988168 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:10.988455 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:10.988542 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:10.990455 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:10.990462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:10.990462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:10.990469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:10.991814 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:10.992222 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:10.992225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:10.992225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:10.992284 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:10.992491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:10.992492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:10.992492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:10.992647 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:10.992739 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:10.992912 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 20:58:10.993004 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":77,"threads_started":1}
I20260619 20:58:10.993700 16408 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b538a6564134d1cbe1d48f1eb7ba2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35026
I20260619 20:58:10.994277 16483 tablet_replica.cc:330] stopping tablet replica
I20260619 20:58:10.994390 16483 raft_consensus.cc:2229] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:10.994462 16483 raft_consensus.cc:2258] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:10.994725 16483 ts_tablet_manager.cc:1905] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:10.996722 16483 ts_tablet_manager.cc:1918] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:10.996807 16483 log.cc:1196] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal/wals/4b538a6564134d1cbe1d48f1eb7ba2bd
I20260619 20:58:10.997016 16483 ts_tablet_manager.cc:1939] T 4b538a6564134d1cbe1d48f1eb7ba2bd P 244d19134f4b41ca90ca9bd48759615b: Deleting consensus metadata
I20260619 20:58:10.997495 16273 catalog_manager.cc:4841] TS 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771): tablet 4b538a6564134d1cbe1d48f1eb7ba2bd (table test-table [id=d363125592cc4a309d0ca4d490f07d54]) successfully deleted
W20260619 20:58:11.061888 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:11.062077 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:11.062099 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:11.063233 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:11.063275 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:11.063308 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:11.064674 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37699
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:11.064896 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:11.065070 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:11.069825 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:11.069875 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:11.069912 16480 server_base.cc:1034] running on GCE node
W20260619 20:58:11.069878 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:11.070137 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:11.070406 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:11.071581 16480 hybrid_clock.cc:648] HybridClock initialized: now 1781902691071567 us; error 33 us; skew 500 ppm
Jun 19 20:58:11 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:11.073948 16480 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:58:11.074452 16480 webserver.cc:460] Webserver started at http://127.0.98.2:37699/ using document root <none> and password file <none>
I20260619 20:58:11.074674 16480 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:11.074756 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:11.076114 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:11.076735 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.076926 16480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:11.077030 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "6fdad759ad5e44018a1768d17e5d29c2"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "e0db66b1840e4b82c91d8b0ef5ec2a30"
server_key_iv: "c321ef648e0be67f8293c00872fbdfbd"
server_key_version: "encryptionkey@0"
I20260619 20:58:11.077368 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:11.097090 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:11.098861 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:11.099036 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:11.099273 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:11.099586 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:11.099640 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.099676 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:11.099702 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.105502 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33237
I20260619 20:58:11.105558 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33237 every 8 connection(s)
I20260619 20:58:11.105852 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:58:11 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:11.114068 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.106456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46355 (local address 127.0.98.62:40069)
0619 20:58:11.106501 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.106504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.106736 (+ 232us) server_negotiation.cc:407] Connection header received
0619 20:58:11.107191 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.107192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.107205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.107258 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:11.108107 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.108581 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.109486 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.109664 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.109711 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.111592 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:11.111593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:11.111594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:11.111600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:11.113017 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.113344 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.113345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.113345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.113390 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.113577 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.113578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.113578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.113675 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:11.113718 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.113910 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.114005 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:58:11.114181 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.106478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:46355)
0619 20:58:11.106614 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.106646 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.107109 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.107308 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.107311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.107844 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:11.108031 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.108035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.108658 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.108659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.109415 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.109417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.109729 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.109730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.109816 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.110252 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:11.110260 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:11.111513 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:11.113082 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:11.113087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:11.113088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:11.113284 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:11.113444 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:11.113444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:11.113444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:11.113528 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:11.113722 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:11.113725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:11.113819 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.113893 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.114011 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":87,"threads_started":1}
I20260619 20:58:11.114667 16612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:58:11.114738 16612 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:11.114950 16612 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:58:11.115368 16286 ts_manager.cc:194] Registered new tserver with Master: 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
I20260619 20:58:11.115574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16480
I20260619 20:58:11.115787 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46355
I20260619 20:58:11.115830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43241
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=42499
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:11.196038 16615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:11.196220 16615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:11.196242 16615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:11.197324 16615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:11.197363 16615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:11.197412 16615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:11.199198 16615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42499
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:11.199431 16615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:11.199695 16615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:58:11.204617 16615 server_base.cc:1034] running on GCE node
W20260619 20:58:11.204577 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:11.204597 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:11.204577 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:11.204980 16615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:11.205195 16615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:11.206378 16615 hybrid_clock.cc:648] HybridClock initialized: now 1781902691206359 us; error 38 us; skew 500 ppm
Jun 19 20:58:11 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:11.208760 16615 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:58:11.209187 16615 webserver.cc:460] Webserver started at http://127.0.98.3:42499/ using document root <none> and password file <none>
I20260619 20:58:11.209403 16615 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:11.209475 16615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:11.210762 16615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:11.211283 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.211447 16615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:11.211529 16615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "479c6c344cc8455fb315a77e8784c82c"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "965022ee40118305350b2b1e0b93e898"
server_key_iv: "9ab31237aedfda811f83aa0b044dfa7f"
server_key_version: "encryptionkey@0"
I20260619 20:58:11.211797 16615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:11.223050 16615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:11.225407 16615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:11.225515 16615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:11.225673 16615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:11.225975 16615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:11.226011 16615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.226064 16615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:11.226114 16615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:11.231884 16615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43241
I20260619 20:58:11.231962 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43241 every 8 connection(s)
I20260619 20:58:11.232280 16615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:58:11 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:11.240489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16615
I20260619 20:58:11.240952 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.232989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47869 (local address 127.0.98.62:40069)
0619 20:58:11.233042 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.233045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.233280 (+ 235us) server_negotiation.cc:407] Connection header received
0619 20:58:11.233713 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.233714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.233723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.233772 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:11.234697 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.235159 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.236054 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.236219 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.236266 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.238094 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:11.238096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:11.238096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:11.238104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:11.239523 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.239942 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.239944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.239944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.240012 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.240261 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.240262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.240262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.240392 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:11.240465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.240772 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.240862 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:58:11.241125 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.232983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:47869)
0619 20:58:11.233162 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.233194 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.233630 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.233823 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.233826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.234434 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:11.234612 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.234617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.235232 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.235233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.235983 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.235984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.236283 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.236284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.236368 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.236752 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:11.236759 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:11.237997 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:11.239638 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:11.239645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:11.239647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:11.239863 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:11.240079 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:11.240080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:11.240080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:11.240187 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:11.240456 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:11.240470 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:11.240629 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.240748 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.240893 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260619 20:58:11.241655 16744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40069
I20260619 20:58:11.241746 16744 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:11.241910 16744 heartbeater.cc:511] Master 127.0.98.62:40069 requested a full tablet report, sending...
I20260619 20:58:11.242352 16286 ts_manager.cc:194] Registered new tserver with Master: 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241)
I20260619 20:58:11.242784 16286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47869
I20260619 20:58:11.246970 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.240875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.62:40069)
0619 20:58:11.241013 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.241014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.241066 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:58:11.241154 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.241155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.241165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.241238 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:11.241998 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.242654 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.243542 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.243724 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.243773 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.244927 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:11.244928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:11.244928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:11.244934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:11.246029 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.246379 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.246380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.246380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.246430 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.246577 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.246577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.246577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.246655 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:11.246756 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.246800 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.246856 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260619 20:58:11.247684 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:11.250449 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.248044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.98.62:40069)
0619 20:58:11.248095 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.248097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.248213 (+ 116us) server_negotiation.cc:407] Connection header received
0619 20:58:11.248297 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.248298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.248336 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.248409 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:11.248504 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.248961 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.249718 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.249910 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.249990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.250155 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:11.250196 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 20:58:11.250386 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59688: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:58:11.250500 16748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59688: token expired
W20260619 20:58:11.250624 16218 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40069)
I20260619 20:58:11.256191 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.250893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.0.98.62:40069)
0619 20:58:11.250964 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.250966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.250972 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:58:11.250989 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.250990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.250997 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.251050 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:11.251500 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.251942 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.252592 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.252782 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.252861 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.253866 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:11.253867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:11.253867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:11.253874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:11.255086 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.255466 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.255467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.255468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.255524 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:11.255714 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:11.255715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:11.255715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:11.255812 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:11.255926 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.256035 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.256103 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 20:58:11.257176 16286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:11.258709 16286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:11.260839 16408 tablet_service.cc:1467] Processing CreateTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 (DEFAULT_TABLE table=test-table [id=88595bd9ac88485b91b340ddfb9b4871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:11.261189 16408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1055259d7ea4fd1a307e6d19ff55280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:11.262913 16366 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Bootstrap starting.
I20260619 20:58:11.263237 16366 tablet_bootstrap.cc:654] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:11.263778 16366 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: No bootstrap required, opened a new log
I20260619 20:58:11.263826 16366 ts_tablet_manager.cc:1397] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:58:11.264151 16366 raft_consensus.cc:348] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.264230 16366 raft_consensus.cc:374] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:11.264276 16366 raft_consensus.cc:729] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d19134f4b41ca90ca9bd48759615b, State: Initialized, Role: FOLLOWER
I20260619 20:58:11.264329 16366 consensus_queue.cc:260] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.264429 16366 ts_tablet_manager.cc:1428] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:58:11.265022 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.260569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.0.1:51038)
0619 20:58:11.260733 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.260743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.260799 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.261278 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.261279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.261281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:11.261321 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.261323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.262653 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.262654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.263725 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.263728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.264741 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.264742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.264807 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.264830 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.264887 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.264923 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260619 20:58:11.265022 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.260569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33237 (local address 127.0.0.1:55662)
0619 20:58:11.260687 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.260697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.260733 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.261431 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.261432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.261434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:11.261490 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.261492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.262967 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.262968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.263900 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.263902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.264771 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.264772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.264838 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.264843 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.264876 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.264923 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5}
I20260619 20:58:11.265311 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.260825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.0.98.3:43241)
0619 20:58:11.261018 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.261023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.261035 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:58:11.261073 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.261075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.261092 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.261172 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:11.261404 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.262573 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.263820 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.264542 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.264620 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.264699 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.265158 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.265217 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260619 20:58:11.265395 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.261042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.2:33237)
0619 20:58:11.261223 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.261228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.261241 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:58:11.261285 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.261287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.261303 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.261630 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:11.261723 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.262834 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.264008 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.264659 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.265059 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.265230 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.265262 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.265300 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260619 20:58:11.266144 16678 tablet_service.cc:1467] Processing CreateTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 (DEFAULT_TABLE table=test-table [id=88595bd9ac88485b91b340ddfb9b4871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:11.266340 16546 tablet_service.cc:1467] Processing CreateTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 (DEFAULT_TABLE table=test-table [id=88595bd9ac88485b91b340ddfb9b4871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:11.266546 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1055259d7ea4fd1a307e6d19ff55280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:11.266669 16546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1055259d7ea4fd1a307e6d19ff55280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:11.269024 16754 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Bootstrap starting.
I20260619 20:58:11.269093 16755 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Bootstrap starting.
I20260619 20:58:11.269721 16755 tablet_bootstrap.cc:654] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:11.269850 16754 tablet_bootstrap.cc:654] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:11.270299 16755 log.cc:824] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:11.270401 16754 log.cc:824] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:11.270927 16755 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: No bootstrap required, opened a new log
I20260619 20:58:11.270978 16755 ts_tablet_manager.cc:1397] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:58:11.271064 16754 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: No bootstrap required, opened a new log
I20260619 20:58:11.271145 16754 ts_tablet_manager.cc:1397] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:58:11.272248 16755 raft_consensus.cc:348] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.272382 16755 raft_consensus.cc:374] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:11.272403 16755 raft_consensus.cc:729] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdad759ad5e44018a1768d17e5d29c2, State: Initialized, Role: FOLLOWER
I20260619 20:58:11.272346 16754 raft_consensus.cc:348] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.272552 16754 raft_consensus.cc:374] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:11.272540 16755 consensus_queue.cc:260] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.272596 16754 raft_consensus.cc:729] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479c6c344cc8455fb315a77e8784c82c, State: Initialized, Role: FOLLOWER
I20260619 20:58:11.272716 16754 consensus_queue.cc:260] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.272801 16755 ts_tablet_manager.cc:1428] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:58:11.272891 16612 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
I20260619 20:58:11.272989 16754 ts_tablet_manager.cc:1428] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:58:11.273073 16744 heartbeater.cc:503] Master 127.0.98.62:40069 was elected leader, sending a full tablet report...
I20260619 20:58:11.343078 16758 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:11.343235 16758 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.343616 16758 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771)
I20260619 20:58:11.346508 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.343786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40771 (local address 127.0.98.2:59611)
0619 20:58:11.343843 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.343856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.343935 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.344109 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.344110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.344112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:11.344185 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.344187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.344884 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.344885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.345654 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.345656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.346274 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.346275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.346346 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.346349 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.346402 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.346440 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":23}
I20260619 20:58:11.346509 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.343906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59611 (local address 127.0.98.1:40771)
0619 20:58:11.343963 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.343965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.343975 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:11.344001 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.344002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.344033 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.344106 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:11.344258 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.344795 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.345753 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.346157 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.346274 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.346352 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.346407 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.346445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:58:11.346589 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.343881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35109 (local address 127.0.98.3:43241)
0619 20:58:11.343927 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.343930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.344123 (+ 193us) server_negotiation.cc:407] Connection header received
0619 20:58:11.344186 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.344187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.344199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.344254 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:11.344402 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.344883 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.345833 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.346256 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.346325 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.346334 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.346489 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.346527 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:58:11.346606 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.343793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.98.2:35109)
0619 20:58:11.344044 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.344053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:11.344127 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:11.344257 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:11.344257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:11.344259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:11.344333 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.344334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.344983 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.344984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.345739 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:11.345741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.346328 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.346329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:11.346399 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.346401 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:11.346481 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:58:11.346522 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":81,"thread_start_us":105,"threads_started":1}
I20260619 20:58:11.346822 16428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1055259d7ea4fd1a307e6d19ff55280" candidate_uuid: "6fdad759ad5e44018a1768d17e5d29c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d19134f4b41ca90ca9bd48759615b" is_pre_election: true
I20260619 20:58:11.346971 16428 raft_consensus.cc:2454] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fdad759ad5e44018a1768d17e5d29c2 in term 0.
I20260619 20:58:11.347025 16698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1055259d7ea4fd1a307e6d19ff55280" candidate_uuid: "6fdad759ad5e44018a1768d17e5d29c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c6c344cc8455fb315a77e8784c82c" is_pre_election: true
I20260619 20:58:11.347179 16698 raft_consensus.cc:2454] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fdad759ad5e44018a1768d17e5d29c2 in term 0.
I20260619 20:58:11.347177 16502 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b, 6fdad759ad5e44018a1768d17e5d29c2; no voters:
I20260619 20:58:11.347326 16758 raft_consensus.cc:2781] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:11.347402 16758 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:11.347426 16758 raft_consensus.cc:3037] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:11.348120 16758 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.348253 16758 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [CANDIDATE]: Term 1 election: Requested vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771)
I20260619 20:58:11.348417 16698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1055259d7ea4fd1a307e6d19ff55280" candidate_uuid: "6fdad759ad5e44018a1768d17e5d29c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c6c344cc8455fb315a77e8784c82c"
I20260619 20:58:11.348498 16698 raft_consensus.cc:3037] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:11.349174 16698 raft_consensus.cc:2454] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fdad759ad5e44018a1768d17e5d29c2 in term 1.
I20260619 20:58:11.349291 16428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1055259d7ea4fd1a307e6d19ff55280" candidate_uuid: "6fdad759ad5e44018a1768d17e5d29c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d19134f4b41ca90ca9bd48759615b"
I20260619 20:58:11.349354 16500 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2; no voters:
I20260619 20:58:11.349385 16428 raft_consensus.cc:3037] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:11.349424 16758 raft_consensus.cc:2781] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:11.349560 16758 raft_consensus.cc:686] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6fdad759ad5e44018a1768d17e5d29c2, State: Running, Role: LEADER
I20260619 20:58:11.349648 16758 consensus_queue.cc:237] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:11.350302 16428 raft_consensus.cc:2454] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fdad759ad5e44018a1768d17e5d29c2 in term 1.
I20260619 20:58:11.350788 16286 catalog_manager.cc:5495] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6fdad759ad5e44018a1768d17e5d29c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } health_report { overall_health: UNKNOWN } } }
I20260619 20:58:11.356266 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.353841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.98.2:33237)
0619 20:58:11.353896 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:11.353910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:11.353918 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:58:11.353948 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:11.353949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:11.353965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:11.354035 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:11.354124 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.354632 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.355636 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:11.355806 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:11.355858 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:11.355964 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:11.356045 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:11.356099 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:11.356130 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:58:11.356145 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 20:58:11.356348 16613 tablet.cc:2369] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:11.357788 16428 raft_consensus.cc:1261] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Refusing update from remote peer 6fdad759ad5e44018a1768d17e5d29c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:11.357795 16698 raft_consensus.cc:1261] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Refusing update from remote peer 6fdad759ad5e44018a1768d17e5d29c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:11.358052 16758 consensus_queue.cc:1035] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:11.358157 16761 consensus_queue.cc:1035] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:11.361256 16766 mvcc.cc:205] Tried to move back new op lower bound from 7298673423799287808 to 7298673423768825856. Current Snapshot: MvccSnapshot[applied={T|T < 7298673423799287808 or (T in {7298673423799287808})}]
I20260619 20:58:11.361378 16765 mvcc.cc:205] Tried to move back new op lower bound from 7298673423799287808 to 7298673423768825856. Current Snapshot: MvccSnapshot[applied={T|T < 7298673423799287808 or (T in {7298673423799287808})}]
I20260619 20:58:11.363190 16286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:11.363274 16286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:11.363925 16286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Sending DeleteTablet for 3 replicas of tablet d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:11.364223 16546 tablet_service.cc:1514] Processing DeleteTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55662
I20260619 20:58:11.364214 16678 tablet_service.cc:1514] Processing DeleteTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51038
I20260619 20:58:11.364418 16408 tablet_service.cc:1514] Processing DeleteTablet for tablet d1055259d7ea4fd1a307e6d19ff55280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35026
I20260619 20:58:11.364446 16774 tablet_replica.cc:330] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: stopping tablet replica
I20260619 20:58:11.364511 16483 tablet_replica.cc:330] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: stopping tablet replica
I20260619 20:58:11.364557 16774 raft_consensus.cc:2229] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:11.364563 16483 raft_consensus.cc:2229] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:11.364619 16483 raft_consensus.cc:2258] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:11.364686 16774 raft_consensus.cc:2258] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:11.364825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16324
I20260619 20:58:11.365499 16774 ts_tablet_manager.cc:1905] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:11.366956 16774 ts_tablet_manager.cc:1918] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:11.367043 16774 log.cc:1196] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:11.367285 16774 ts_tablet_manager.cc:1939] T d1055259d7ea4fd1a307e6d19ff55280 P 6fdad759ad5e44018a1768d17e5d29c2: Deleting consensus metadata
I20260619 20:58:11.367640 16272 catalog_manager.cc:4841] TS 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): tablet d1055259d7ea4fd1a307e6d19ff55280 (table test-table [id=88595bd9ac88485b91b340ddfb9b4871]) successfully deleted
I20260619 20:58:11.370344 16775 tablet_replica.cc:330] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: stopping tablet replica
I20260619 20:58:11.370453 16775 raft_consensus.cc:2229] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:11.370553 16775 raft_consensus.cc:2258] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:58:11.370754 16273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40771 (error 108)
I20260619 20:58:11.370939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16480
I20260619 20:58:11.371513 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.371057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40771 (local address 127.0.0.1:35042)
0619 20:58:11.371436 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.371478 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
W20260619 20:58:11.371763 16273 catalog_manager.cc:4584] TS 244d19134f4b41ca90ca9bd48759615b (127.0.98.1:40771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1055259d7ea4fd1a307e6d19ff55280: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40771: connect: Connection refused (error 111)
I20260619 20:58:11.374542 16775 ts_tablet_manager.cc:1905] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:11.375887 16775 ts_tablet_manager.cc:1918] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:11.375945 16775 log.cc:1196] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:11.376116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16615
I20260619 20:58:11.382460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16253
I20260619 20:58:11.382488 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:11.382344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.0.1:51048)
0619 20:58:11.382416 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:11.382450 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 20:58:31.387674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--webserver_interface=127.0.98.62
--webserver_port=40797
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:31.466711 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:31.466905 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:31.466928 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:31.468045 16777 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:58:31.468086 16777 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:58:31.468098 16777 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:58:31.468142 16777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:31.468166 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:31.468191 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:31.468225 16777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:31.468240 16777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:31.469712 16777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:31.469947 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:31.470170 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:31.474759 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.474756 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.474858 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.474934 16777 server_base.cc:1034] running on GCE node
I20260619 20:58:31.475288 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:31.475528 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:31.476661 16777 hybrid_clock.cc:648] HybridClock initialized: now 1781902711476648 us; error 29 us; skew 500 ppm
Jun 19 20:58:31 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:31.479071 16777 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:58:31.479523 16777 webserver.cc:460] Webserver started at http://127.0.98.62:40797/ using document root <none> and password file <none>
I20260619 20:58:31.479755 16777 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:31.479801 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:31.481071 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:31.481760 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.481988 16777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:31.482082 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "119559b53ae54004b38754ebb7425fff"
format_stamp: "Formatted at 2026-06-19 20:57:49 on dist-test-slave-8bgz"
server_key: "8dd800ad3ab805c18d63b13e97ac9e36"
server_key_iv: "372d1c780c8c69ac921651e0ae03c697"
server_key_version: "encryptionkey@0"
I20260619 20:58:31.482463 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:31.492529 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:31.494313 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:31.494453 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:31.498330 16777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40069
I20260619 20:58:31.498365 16844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40069 every 8 connection(s)
I20260619 20:58:31.498715 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:31.500161 16845 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:58:31.500842 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap starting.
I20260619 20:58:31.502328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16777
I20260619 20:58:31.502573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40771
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=33839
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:58:31.509680 16845 log.cc:824] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:31.511965 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:58:31.512279 16845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Bootstrap complete.
I20260619 20:58:31.514801 16845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:31.515034 16845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Initialized, Role: FOLLOWER
I20260619 20:58:31.515239 16845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:31.515321 16845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:31.515383 16845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:31.515435 16845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:58:31.516455 16845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:31.516544 16845 leader_election.cc:302] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119559b53ae54004b38754ebb7425fff; no voters:
I20260619 20:58:31.516745 16845 leader_election.cc:288] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:58:31.516783 16849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:58:31.516902 16849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [term 3 LEADER]: Becoming Leader. State: Replica: 119559b53ae54004b38754ebb7425fff, State: Running, Role: LEADER
I20260619 20:58:31.517041 16845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:31.517055 16849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260619 20:58:31.517396 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119559b53ae54004b38754ebb7425fff. Latest consensus state: current_term: 3 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:58:31.517565 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:31.517398 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "119559b53ae54004b38754ebb7425fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119559b53ae54004b38754ebb7425fff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260619 20:58:31.517834 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:31.518451 16856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:31.519486 16856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:31.519627 16856 catalog_manager.cc:1259] Loaded cluster ID: c7ea86091a2448afaff6c92cc7165efb
I20260619 20:58:31.519662 16856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:31.520977 16856 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:31.521169 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Loaded TSK: 0
I20260619 20:58:31.521222 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 119559b53ae54004b38754ebb7425fff: Loaded TSK: 1
F20260619 20:58:31.521468 16856 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f15e79649ad google::LogMessage::Fail() at ??:0
@ 0x7f15e796788a google::LogMessage::SendToLog() at ??:0
@ 0x7f15e79644de google::LogMessage::Flush() at ??:0
@ 0x7f15e7965a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f15ea0d1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f15ea0ced41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f15ef8be1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f15ef8be48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f15ef8be594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f15ef884c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f15ef88ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f15e98320d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f15e9827954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f15ea68b6db start_thread at ??:0
@ 0x7f15e646261f clone at ??:0
W20260619 20:58:31.604511 16847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:31.604710 16847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:31.604735 16847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:31.605969 16847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:31.606011 16847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:31.606040 16847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:31.607615 16847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33839
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:31.607853 16847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:31.608093 16847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:31.613195 16887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.613248 16886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.613382 16847 server_base.cc:1034] running on GCE node
W20260619 20:58:31.613543 16889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.613754 16847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:31.614055 16847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:31.615211 16847 hybrid_clock.cc:648] HybridClock initialized: now 1781902711615180 us; error 49 us; skew 500 ppm
Jun 19 20:58:31 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:31.617805 16847 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:58:31.618366 16847 webserver.cc:460] Webserver started at http://127.0.98.1:33839/ using document root <none> and password file <none>
I20260619 20:58:31.618595 16847 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:31.618665 16847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:31.620162 16847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:58:31.620906 16896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.621124 16847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:31.621222 16847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "244d19134f4b41ca90ca9bd48759615b"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "742f1dd7ccbe35513fd250f93bff4866"
server_key_iv: "1305f27304edffa58aaa0b6405853a2a"
server_key_version: "encryptionkey@0"
I20260619 20:58:31.621578 16847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:31.636916 16847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:31.639308 16847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:31.639489 16847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:31.639745 16847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:31.640187 16903 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:58:31.642280 16847 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:58:31.642343 16847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:58:31.642403 16847 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:58:31.643266 16847 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:58:31.643332 16847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:58:31.643388 16903 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Bootstrap starting.
I20260619 20:58:31.650480 16847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40771
I20260619 20:58:31.650635 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40771 every 8 connection(s)
I20260619 20:58:31.650897 16847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:31.654134 16903 log.cc:824] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:31.656725 16903 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:58:31.657054 16903 tablet_bootstrap.cc:492] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Bootstrap complete.
I20260619 20:58:31.657528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 16847
I20260619 20:58:31.657778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33237
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=37699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:58:31.658327 16903 ts_tablet_manager.cc:1397] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260619 20:58:31.659704 16903 raft_consensus.cc:348] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:31.660024 16903 raft_consensus.cc:729] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d19134f4b41ca90ca9bd48759615b, State: Initialized, Role: FOLLOWER
I20260619 20:58:31.660158 16903 consensus_queue.cc:260] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:31.660401 16903 ts_tablet_manager.cc:1428] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260619 20:58:31.711788 17013 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:58:31.651510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:50507)
0619 20:58:31.651669 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.651710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:31.652147 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:31.711583 (+ 59436us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260619 20:58:31.712086 17011 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40069 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40069: Client connection negotiation failed: client connection to 127.0.98.62:40069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:58:31.712432 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.712314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:39509)
0619 20:58:31.712360 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.712407 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 20:58:31.712795 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.712698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:34895)
0619 20:58:31.712740 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.712765 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:58:31.712941 17011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:58:31.739470 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:31.739655 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:31.739678 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:31.740800 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:31.740840 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:31.740895 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:31.742355 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37699
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:31.742581 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:31.742797 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:31.747644 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.747691 17015 server_base.cc:1034] running on GCE node
W20260619 20:58:31.747699 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.747659 17021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.748052 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:31.748282 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:31.749439 17015 hybrid_clock.cc:648] HybridClock initialized: now 1781902711749430 us; error 35 us; skew 500 ppm
Jun 19 20:58:31 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:31.751812 17015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:58:31.752161 17015 webserver.cc:460] Webserver started at http://127.0.98.2:37699/ using document root <none> and password file <none>
I20260619 20:58:31.752394 17015 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:31.752439 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:31.753618 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:31.754136 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.754320 17015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.754403 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "6fdad759ad5e44018a1768d17e5d29c2"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "e0db66b1840e4b82c91d8b0ef5ec2a30"
server_key_iv: "c321ef648e0be67f8293c00872fbdfbd"
server_key_version: "encryptionkey@0"
I20260619 20:58:31.754719 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:31.784750 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:31.786787 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:31.786966 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:31.787202 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:31.787514 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:31.787565 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.787601 17015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:31.787628 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.793319 17015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33237
I20260619 20:58:31.793411 17144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33237 every 8 connection(s)
I20260619 20:58:31.793722 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:31.794847 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.794487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:58271)
0619 20:58:31.794640 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.794710 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260619 20:58:31.795258 17145 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40069 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40069: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
I20260619 20:58:31.795506 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.795423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:40301)
0619 20:58:31.795464 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.795489 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 20:58:31.795745 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.795680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:59299)
0619 20:58:31.795713 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.795733 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 20:58:31.795828 17145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:58:31.802958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17015
I20260619 20:58:31.803225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43241
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40069
--webserver_port=42499
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:31.883383 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:31.883576 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:31.883599 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:31.884743 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:31.884788 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:31.884845 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:31.886356 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42499
--tserver_master_addrs=127.0.98.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:31.886587 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:31.886806 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:31.891637 17153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.891614 17156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:31.891615 17154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:31.891825 17148 server_base.cc:1034] running on GCE node
I20260619 20:58:31.892027 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:31.892274 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:31.893422 17148 hybrid_clock.cc:648] HybridClock initialized: now 1781902711893395 us; error 41 us; skew 500 ppm
Jun 19 20:58:31 dist-test-slave-8bgz krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:31.895893 17148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:58:31.896299 17148 webserver.cc:460] Webserver started at http://127.0.98.3:42499/ using document root <none> and password file <none>
I20260619 20:58:31.896525 17148 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:31.896589 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:31.897921 17148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:31.898550 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.898708 17148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.898800 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "479c6c344cc8455fb315a77e8784c82c"
format_stamp: "Formatted at 2026-06-19 20:57:50 on dist-test-slave-8bgz"
server_key: "965022ee40118305350b2b1e0b93e898"
server_key_iv: "9ab31237aedfda811f83aa0b044dfa7f"
server_key_version: "encryptionkey@0"
I20260619 20:58:31.899153 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:31.912804 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:31.914760 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:31.914933 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:31.915153 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:31.915560 17170 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 20:58:31.916884 17170 ts_tablet_manager.cc:1852] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 20:58:31.917133 17170 ts_tablet_manager.cc:1905] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:31.919183 17170 ts_tablet_manager.cc:1918] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:31.919275 17170 ts_tablet_manager.cc:1939] T d1055259d7ea4fd1a307e6d19ff55280 P 479c6c344cc8455fb315a77e8784c82c: Deleting consensus metadata
I20260619 20:58:31.919461 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 20:58:31.919528 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 20:58:31.919571 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:31.919597 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:31.925549 17148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43241
I20260619 20:58:31.925617 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43241 every 8 connection(s)
I20260619 20:58:31.925906 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:31.926885 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.926534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:42679)
0619 20:58:31.926690 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.926756 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
W20260619 20:58:31.927086 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40069 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40069: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
I20260619 20:58:31.927326 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.927245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:34805)
0619 20:58:31.927284 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.927309 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 20:58:31.927563 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:31.927515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:60047)
0619 20:58:31.927535 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:31.927552 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260619 20:58:31.927644 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:58:31.927906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17148
I20260619 20:58:32.713474 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:32.713250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:45515)
0619 20:58:32.713371 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:32.713406 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260619 20:58:32.796342 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:32.796124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:40097)
0619 20:58:32.796238 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:32.796270 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260619 20:58:32.928078 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:32.927907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:42893)
0619 20:58:32.928010 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:32.928037 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260619 20:58:33.093597 17285 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:33.093713 17285 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:33.093986 17285 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
Jun 19 20:58:33 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 20:58:33 dist-test-slave-8bgz krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 20:58:33.107347 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.094175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33237 (local address 127.0.98.1:43601)
0619 20:58:33.094601 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:33.094610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:33.094656 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:33.095532 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:33.095535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:33.096058 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:33.096329 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:33.096337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.098004 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.098004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:33.099276 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:33.099279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.100146 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.100148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:33.100297 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:33.100941 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:33.100964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:33.103230 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:33.105289 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:33.105290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:33.105290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:33.105747 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:33.106172 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:33.106173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:33.106173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:33.106266 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:33.106735 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:33.106741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:33.106899 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 20:58:33.107021 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 20:58:33.107207 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":364,"thread_start_us":48,"threads_started":1}
I20260619 20:58:33.107357 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.094329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43601 (local address 127.0.98.2:33237)
0619 20:58:33.094619 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:33.094627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:33.094657 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:58:33.095213 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:33.095217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:33.095296 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:33.095458 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:33.096465 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.097877 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.099384 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.100040 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.100173 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:33.103334 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:33.103338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:33.103339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:33.103351 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:33.105212 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:33.105990 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:33.105992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:33.105992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:33.106106 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:33.106341 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:33.106342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:33.106342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:33.106661 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:33.106748 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:33.107107 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 20:58:33.107245 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
I20260619 20:58:33.107357 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.094366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49053 (local address 127.0.98.3:43241)
0619 20:58:33.094536 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:33.094549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:33.094577 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:58:33.095264 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:33.095268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:33.095311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:33.095458 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:33.096465 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.097908 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.099384 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.100044 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.100167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:33.102991 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:33.102996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:33.102996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:33.103009 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:33.105164 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:33.105883 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:33.105886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:33.105886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:33.106024 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:33.106341 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:33.106342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:33.106342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:33.106661 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:33.106748 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:33.107108 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 20:58:33.107244 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":53,"threads_started":1}
I20260619 20:58:33.107404 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.094176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43241 (local address 127.0.98.1:49053)
0619 20:58:33.094252 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:33.094265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:33.094347 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:33.095532 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:33.095535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:33.096058 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:33.096329 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:33.096341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.098000 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.098001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:33.099276 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:33.099279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:33.100148 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:33.100149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:33.100297 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:33.100993 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:33.100997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:33.102874 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:33.105249 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:33.105255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:33.105269 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:33.105747 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:33.106099 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:33.106099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:33.106099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:33.106266 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:33.106735 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:33.106741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:33.106899 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 20:58:33.107021 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 20:58:33.107261 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":172}
W20260619 20:58:33.108677 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:33.108764 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:33.108817 16898 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:33.108917 17285 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:33.714295 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.714032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:52201)
0619 20:58:33.714159 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:33.714193 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260619 20:58:33.797019 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.796804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:48935)
0619 20:58:33.796934 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:33.796973 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260619 20:58:33.928633 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:33.928453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:43755)
0619 20:58:33.928565 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:33.928597 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260619 20:58:34.714942 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:34.714731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:50163)
0619 20:58:34.714865 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:34.714898 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260619 20:58:34.797830 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:34.797630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:42715)
0619 20:58:34.797744 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:34.797778 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260619 20:58:34.929211 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:34.929033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:33189)
0619 20:58:34.929148 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:34.929177 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260619 20:58:34.980973 17295 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:34.981073 17295 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:34.981252 17295 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
W20260619 20:58:34.981633 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:34.981720 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:34.981788 16899 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:34.981952 17295 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:35.715574 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:35.715369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:53043)
0619 20:58:35.715490 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:35.715527 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260619 20:58:35.798530 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:35.798313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:59183)
0619 20:58:35.798443 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:35.798482 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260619 20:58:35.929804 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:35.929612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:53079)
0619 20:58:35.929736 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:35.929769 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260619 20:58:36.716248 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:36.716041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:38903)
0619 20:58:36.716167 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:36.716202 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260619 20:58:36.799201 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:36.799001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:58953)
0619 20:58:36.799119 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:36.799154 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260619 20:58:36.930413 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:36.930183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:47893)
0619 20:58:36.930340 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:36.930374 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260619 20:58:37.559147 17302 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:37.559234 17302 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:37.559444 17302 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
W20260619 20:58:37.559804 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:37.559863 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:37.559890 16899 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:37.559949 17302 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:37.717033 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:37.716833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:58791)
0619 20:58:37.716953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:37.716988 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260619 20:58:37.799822 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:37.799626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:38333)
0619 20:58:37.799746 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:37.799776 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260619 20:58:37.931267 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:37.931046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:45821)
0619 20:58:37.931205 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:37.931234 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260619 20:58:38.717758 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:38.717472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:41875)
0619 20:58:38.717616 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:38.717695 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260619 20:58:38.800424 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:38.800214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:42723)
0619 20:58:38.800341 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:38.800378 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260619 20:58:38.931928 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:38.931746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:59787)
0619 20:58:38.931862 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:38.931895 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260619 20:58:39.718407 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:39.718169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:37577)
0619 20:58:39.718330 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:39.718360 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260619 20:58:39.801034 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:39.800838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:49465)
0619 20:58:39.800958 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:39.800989 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260619 20:58:39.932574 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:39.932312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:40235)
0619 20:58:39.932474 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:39.932527 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260619 20:58:40.719045 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:40.718826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:48381)
0619 20:58:40.718964 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:40.718999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260619 20:58:40.801700 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:40.801496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:44635)
0619 20:58:40.801621 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:40.801652 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260619 20:58:40.829137 17314 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:40.829223 17314 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:40.829443 17314 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
W20260619 20:58:40.829823 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:40.829907 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:40.829968 16898 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:40.830056 17314 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:40.933234 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:40.933047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:57021)
0619 20:58:40.933166 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:40.933198 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260619 20:58:41.719800 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:41.719562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:55461)
0619 20:58:41.719711 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:41.719745 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260619 20:58:41.802515 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:41.802300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:55493)
0619 20:58:41.802426 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:41.802468 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260619 20:58:41.933982 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:41.933778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:42217)
0619 20:58:41.933907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:41.933943 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260619 20:58:42.701306 17319 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:42.701388 17319 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:42.701591 17319 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
W20260619 20:58:42.701936 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:42.702021 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:42.702096 16899 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:42.702169 17319 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:42.720541 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:42.720246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:53195)
0619 20:58:42.720455 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:42.720491 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260619 20:58:42.803120 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:42.802932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:40323)
0619 20:58:42.803039 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:42.803075 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20260619 20:58:42.934612 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:42.934442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:56465)
0619 20:58:42.934547 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:42.934578 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260619 20:58:43.721143 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:43.720937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:44073)
0619 20:58:43.721061 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:43.721099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260619 20:58:43.803720 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:43.803526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:50945)
0619 20:58:43.803637 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:43.803674 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260619 20:58:43.935250 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:43.935054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:39561)
0619 20:58:43.935179 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:43.935213 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260619 20:58:44.721873 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:44.721649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:50617)
0619 20:58:44.721790 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:44.721828 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260619 20:58:44.804318 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:44.804117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:50831)
0619 20:58:44.804228 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:44.804274 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260619 20:58:44.935802 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:44.935623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:59485)
0619 20:58:44.935736 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:44.935768 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260619 20:58:45.722548 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:45.722348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:53685)
0619 20:58:45.722467 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:45.722502 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260619 20:58:45.804922 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:45.804699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:37057)
0619 20:58:45.804840 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:45.804877 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260619 20:58:45.936408 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:45.936219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:37237)
0619 20:58:45.936340 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:45.936372 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260619 20:58:46.362874 17332 raft_consensus.cc:482] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:46.362979 17332 raft_consensus.cc:504] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479c6c344cc8455fb315a77e8784c82c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43241 } } peers { permanent_uuid: "6fdad759ad5e44018a1768d17e5d29c2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33237 } } peers { permanent_uuid: "244d19134f4b41ca90ca9bd48759615b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40771 } }
I20260619 20:58:46.363171 17332 leader_election.cc:288] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241), 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237)
W20260619 20:58:46.363554 16898 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 479c6c344cc8455fb315a77e8784c82c (127.0.98.3:43241): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
W20260619 20:58:46.363646 16899 leader_election.cc:341] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fdad759ad5e44018a1768d17e5d29c2 (127.0.98.2:33237): Not found: Tablet not found: d1055259d7ea4fd1a307e6d19ff55280
I20260619 20:58:46.363692 16899 leader_election.cc:302] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d19134f4b41ca90ca9bd48759615b; no voters: 479c6c344cc8455fb315a77e8784c82c, 6fdad759ad5e44018a1768d17e5d29c2
I20260619 20:58:46.363773 17332 raft_consensus.cc:2726] T d1055259d7ea4fd1a307e6d19ff55280 P 244d19134f4b41ca90ca9bd48759615b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:58:46.723367 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:46.723091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.1:34571)
0619 20:58:46.723250 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:46.723303 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260619 20:58:46.805577 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:46.805353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.2:51187)
0619 20:58:46.805482 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:46.805517 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260619 20:58:46.932744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 16847
I20260619 20:58:46.937100 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:46.936879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40069 (local address 127.0.98.3:39215)
0619 20:58:46.937009 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:46.937048 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":49,"threads_started":1}
I20260619 20:58:46.937366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17015
I20260619 20:58:46.941744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17148
2026-06-19T20:58:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58187 ms)
[----------] 30 tests from SecurityITest (229830 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:46 dist-test-slave-8bgz krb5kdc[17338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:46 dist-test-slave-8bgz krb5kdc[17338](info): set up 2 sockets
Jun 19 20:58:46 dist-test-slave-8bgz krb5kdc[17338](info): commencing operation
krb5kdc: starting...
W20260619 20:58:47.973948 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:58:48.085314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:48.163221 17351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:48.163391 17351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:48.163412 17351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:48.164557 17351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:48.164597 17351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:48.164610 17351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:48.164623 17351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:48.164633 17351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:48.166023 17351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:48.166273 17351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:48.166530 17351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:48.171506 17357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.171612 17351 server_base.cc:1034] running on GCE node
W20260619 20:58:48.171550 17356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.171512 17359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.171993 17351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:48.172302 17351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:48.173458 17351 hybrid_clock.cc:648] HybridClock initialized: now 1781902728173444 us; error 42 us; skew 500 ppm
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:48.175886 17351 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:58:48.176285 17351 webserver.cc:460] Webserver started at http://127.0.98.62:33633/ using document root <none> and password file <none>
I20260619 20:58:48.176529 17351 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:48.176622 17351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:48.176774 17351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:48.177704 17351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "32742dfa9c4e46178ef1c26f6d0b8758"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "63cbfc99e74400e87af67b78fe409d39"
server_key_iv: "d581441420947f91dd44d464f3df58f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.178071 17351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "32742dfa9c4e46178ef1c26f6d0b8758"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "63cbfc99e74400e87af67b78fe409d39"
server_key_iv: "d581441420947f91dd44d464f3df58f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.179329 17351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:48.180001 17366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.180157 17351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:48.180251 17351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "32742dfa9c4e46178ef1c26f6d0b8758"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "63cbfc99e74400e87af67b78fe409d39"
server_key_iv: "d581441420947f91dd44d464f3df58f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.180330 17351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:48.196511 17351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:48.198385 17351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:48.198542 17351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:48.202114 17351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44061
I20260619 20:58:48.202148 17418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44061 every 8 connection(s)
I20260619 20:58:48.202510 17351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:48.203305 17419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:48.205390 17419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: Bootstrap starting.
I20260619 20:58:48.205931 17419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:48.206198 17419 log.cc:824] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:48.206914 17419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: No bootstrap required, opened a new log
I20260619 20:58:48.208480 17419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260619 20:58:48.208606 17419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:48.208626 17419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32742dfa9c4e46178ef1c26f6d0b8758, State: Initialized, Role: FOLLOWER
I20260619 20:58:48.208798 17419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260619 20:58:48.208873 17419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:48.208894 17419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:48.208954 17419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:48.209456 17419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260619 20:58:48.209551 17419 leader_election.cc:302] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32742dfa9c4e46178ef1c26f6d0b8758; no voters:
I20260619 20:58:48.209694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17351
I20260619 20:58:48.209766 17419 leader_election.cc:288] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:48.209812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:48.209789 17422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:48.209997 17422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [term 1 LEADER]: Becoming Leader. State: Replica: 32742dfa9c4e46178ef1c26f6d0b8758, State: Running, Role: LEADER
I20260619 20:58:48.210019 392 external_mini_cluster.cc:1428] Setting key 49e1d6b3cd6e2ac250dc5152d46ab713
I20260619 20:58:48.210071 17419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:48.210136 17422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260619 20:58:48.210623 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32742dfa9c4e46178ef1c26f6d0b8758. Latest consensus state: current_term: 1 leader_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } } }
I20260619 20:58:48.210693 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32742dfa9c4e46178ef1c26f6d0b8758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } } }
I20260619 20:58:48.210922 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:48.210712 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:48.211665 17434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:48.212186 17434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:48.213526 17434 catalog_manager.cc:1347] Generated new cluster ID: 232ea533840f447099380724b921a17f
I20260619 20:58:48.213582 17434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:48.221573 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.210492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.0.98.62:44061)
0619 20:58:48.211043 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.211069 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.211102 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:58:48.211210 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.211216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.211265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.211397 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:48.212327 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.213635 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.214635 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.215283 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.215398 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.218022 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:48.218025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:48.218025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:48.218036 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:48.220012 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.220411 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.220414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.220414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.220517 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.220722 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.220723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.220723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.220992 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:48.221143 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.221244 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.221364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":64,"threads_started":1}
I20260619 20:58:48.230408 17434 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:48.230898 17434 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:48.236434 17434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: Generated new TSK 0
I20260619 20:58:48.236580 17434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:58:48.265368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:48.345270 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:48.345458 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:48.345487 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:48.346583 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:48.346623 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:48.346652 17445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:48.348071 17445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:48.348250 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:48.348417 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:48.353412 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.353413 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.353468 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.353806 17445 server_base.cc:1034] running on GCE node
I20260619 20:58:48.353971 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:48.354138 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:48.355268 17445 hybrid_clock.cc:648] HybridClock initialized: now 1781902728355257 us; error 25 us; skew 500 ppm
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:48.357667 17445 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:58:48.358023 17445 webserver.cc:460] Webserver started at http://127.0.98.1:34435/ using document root <none> and password file <none>
I20260619 20:58:48.358301 17445 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:48.358369 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:48.358506 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:48.359467 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0f4ac83a2b74785a0c60e20f06774ad"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "c31b5491db1a16675e6ead64015e723d"
server_key_iv: "8395d3916d94006b8e58915027f8b279"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.359813 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f4ac83a2b74785a0c60e20f06774ad"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "c31b5491db1a16675e6ead64015e723d"
server_key_iv: "8395d3916d94006b8e58915027f8b279"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.361096 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:48.361821 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.362001 17445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:48.362094 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "d0f4ac83a2b74785a0c60e20f06774ad"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "c31b5491db1a16675e6ead64015e723d"
server_key_iv: "8395d3916d94006b8e58915027f8b279"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.362174 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:48.377903 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:48.380156 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:48.380312 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:48.380522 17445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:48.380817 17445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:48.380869 17445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.380909 17445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:48.380936 17445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.387915 17445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44265
I20260619 20:58:48.387993 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44265 every 8 connection(s)
I20260619 20:58:48.388299 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:48.389654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17445
I20260619 20:58:48.389763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:48.389963 392 external_mini_cluster.cc:1428] Setting key e9317ebbf1303c4d7444874e2b745817
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:48.399230 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.389025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55727 (local address 127.0.98.62:44061)
0619 20:58:48.389070 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.389073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.389343 (+ 270us) server_negotiation.cc:407] Connection header received
0619 20:58:48.391161 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.391163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.391177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.391236 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:48.392070 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.392779 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.393762 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.394089 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.394166 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.396239 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:48.396241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:48.396242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:48.396250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:48.397854 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.398245 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.398247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.398247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.398305 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.398576 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.398577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.398577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.398726 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:48.398798 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.399037 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.399126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:58:48.399268 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.389011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.1:55727)
0619 20:58:48.389179 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.389220 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.391026 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.391306 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.391310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.391681 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:48.391942 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.391948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.392870 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.392870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.393672 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.393674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.394180 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.394181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.394357 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.394868 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:48.394877 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:48.396152 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:48.397942 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.397948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.397950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.398160 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.398408 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.398409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.398409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.398502 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.398798 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:48.398803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:48.398909 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.398983 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.399094 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:58:48.400029 17574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260619 20:58:48.400134 17574 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:48.400302 17574 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260619 20:58:48.400837 17383 ts_manager.cc:194] Registered new tserver with Master: d0f4ac83a2b74785a0c60e20f06774ad (127.0.98.1:44265)
I20260619 20:58:48.401448 17383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:58:48.414534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:48.493018 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:48.493225 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:48.493248 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:48.494434 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:48.494477 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:48.494503 17579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:48.495987 17579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:48.496235 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:48.496467 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:48.501297 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.501367 17579 server_base.cc:1034] running on GCE node
W20260619 20:58:48.501349 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.501327 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.501708 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:48.501909 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:48.503065 17579 hybrid_clock.cc:648] HybridClock initialized: now 1781902728503054 us; error 27 us; skew 500 ppm
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:48.505501 17579 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:58:48.505911 17579 webserver.cc:460] Webserver started at http://127.0.98.2:45099/ using document root <none> and password file <none>
I20260619 20:58:48.506136 17579 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:48.506202 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:48.506381 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:48.507354 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d30c25ca256449e8498a3f87f60427b"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "046eee01de450a04ebb9b141cc438caa"
server_key_iv: "f5000222863ffe263ff7600c3c255113"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.507686 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d30c25ca256449e8498a3f87f60427b"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "046eee01de450a04ebb9b141cc438caa"
server_key_iv: "f5000222863ffe263ff7600c3c255113"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.508997 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:48.509776 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.509959 17579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:48.510068 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "4d30c25ca256449e8498a3f87f60427b"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "046eee01de450a04ebb9b141cc438caa"
server_key_iv: "f5000222863ffe263ff7600c3c255113"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.510142 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:48.530961 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:48.533169 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:48.533278 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:48.533443 17579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:48.533773 17579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:48.533808 17579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.533830 17579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:48.533874 17579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.539508 17579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35557
I20260619 20:58:48.539572 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35557 every 8 connection(s)
I20260619 20:58:48.539852 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:48.548321 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.540582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50905 (local address 127.0.98.62:44061)
0619 20:58:48.540695 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.540698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.540860 (+ 162us) server_negotiation.cc:407] Connection header received
0619 20:58:48.541737 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.541738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.541748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.541805 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:48.542385 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.542893 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.543828 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.544007 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.544058 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.546048 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:48.546049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:48.546049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:48.546055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:48.547239 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.547596 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.547596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.547597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.547645 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.547848 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.547848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.547849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.547941 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:48.547989 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.548195 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.548257 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:58:48.548491 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.540555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.2:50905)
0619 20:58:48.540741 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.540774 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.541626 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.541857 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.541860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.542092 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:48.542298 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.542302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.542970 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.542971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.543752 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.543754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.544077 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.544077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.544173 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.544656 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:48.544664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:48.545965 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:48.547309 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.547314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.547315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.547516 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.547700 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.547700 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.547700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.547796 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.547992 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:48.547995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:48.548095 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.548171 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.548279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
I20260619 20:58:48.548776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17579
I20260619 20:58:48.548897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:48.549005 17708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260619 20:58:48.549104 17708 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:48.549132 392 external_mini_cluster.cc:1428] Setting key 2e44c42bf46f202ec1939b6be669a680
I20260619 20:58:48.549273 17708 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260619 20:58:48.549729 17382 ts_manager.cc:194] Registered new tserver with Master: 4d30c25ca256449e8498a3f87f60427b (127.0.98.2:35557)
I20260619 20:58:48.550248 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50905
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:58:48.571005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:48.649071 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:48.649283 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:48.649318 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:48.650554 17713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:48.650629 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:48.650676 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:48.652136 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:48.652374 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:48.652582 17713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:48.657371 17718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.657361 17721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.657387 17719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.657703 17713 server_base.cc:1034] running on GCE node
I20260619 20:58:48.657907 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:48.658128 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:48.659287 17713 hybrid_clock.cc:648] HybridClock initialized: now 1781902728659277 us; error 69 us; skew 500 ppm
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:48.661630 17713 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:58:48.662010 17713 webserver.cc:460] Webserver started at http://127.0.98.3:34481/ using document root <none> and password file <none>
I20260619 20:58:48.662285 17713 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:48.662351 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:48.662490 17713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:48.663443 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a3fd0f2e5384f138005e5b71174484a"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "b9a7cbaf5bb34d4a1f8258025fc35c90"
server_key_iv: "8ec51977856d16554a04faf0bcb1e659"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.663781 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a3fd0f2e5384f138005e5b71174484a"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "b9a7cbaf5bb34d4a1f8258025fc35c90"
server_key_iv: "8ec51977856d16554a04faf0bcb1e659"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.665125 17713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:48.665804 17728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.665951 17713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:48.666034 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "0a3fd0f2e5384f138005e5b71174484a"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "b9a7cbaf5bb34d4a1f8258025fc35c90"
server_key_iv: "8ec51977856d16554a04faf0bcb1e659"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.666109 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:48.692881 17713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:48.695179 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:48.695343 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:48.695562 17713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:48.695853 17713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:48.695904 17713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.695942 17713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:48.695968 17713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.701730 17713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39363
I20260619 20:58:48.701769 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39363 every 8 connection(s)
I20260619 20:58:48.702101 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:48.705008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17713
I20260619 20:58:48.705123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:48.705344 392 external_mini_cluster.cc:1428] Setting key 938de1857199676035a8722875e976ba
Jun 19 20:58:48 dist-test-slave-8bgz krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:48.711521 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.702830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57029 (local address 127.0.98.62:44061)
0619 20:58:48.702910 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.702912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.703058 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:58:48.703924 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.703925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.703934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.703996 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:48.704545 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.705215 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.706529 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.706795 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.706869 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.708701 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:48.708702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:48.708702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:48.708708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:48.710093 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.710499 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.710500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.710501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.710554 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.710879 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.710880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.710881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.710992 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:48.711065 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.711359 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.711433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 20:58:48.711728 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.702785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.3:57029)
0619 20:58:48.702938 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.702977 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.703841 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.704050 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.704053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.704296 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:48.704467 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.704471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.705311 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.705312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.706452 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.706467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.706873 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.706874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.706966 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.707350 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:48.707358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:48.708627 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:48.710155 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.710159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.710161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.710406 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.710675 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:48.710676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:48.710676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:48.710809 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:48.711075 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:48.711082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:48.711233 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.711337 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.711485 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260619 20:58:48.712152 17842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260619 20:58:48.712226 17842 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:48.712347 17842 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260619 20:58:48.712639 17382 ts_manager.cc:194] Registered new tserver with Master: 0a3fd0f2e5384f138005e5b71174484a (127.0.98.3:39363)
I20260619 20:58:48.713070 17382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57029
I20260619 20:58:48.717105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:48.724493 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.718533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.62:44061)
0619 20:58:48.718589 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.718591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.718709 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:58:48.718788 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.718789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.718799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.718858 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:48.719490 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.719955 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.720725 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.720928 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.721017 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.722169 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:48.722171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:48.722171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:48.722178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:48.723345 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.723727 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.723728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.723729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.723785 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:48.723979 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:48.723980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:48.723980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:48.724117 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:48.724225 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.724328 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.724398 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:58:48.725909 17382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:48.727437 17382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:48.733327 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44265 (local address 127.0.0.1:46068)
0619 20:58:48.729612 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.729647 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.729710 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.730110 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.730111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.730112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:48.730153 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.730154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.731013 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.731014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.732179 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.732181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.733064 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.733139 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733153 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.733212 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.733259 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":2,"thread_start_us":48,"threads_started":1}
I20260619 20:58:48.733353 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.0.98.1:44265)
0619 20:58:48.729897 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.729901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.729910 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:58:48.729944 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.729946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.729962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.730032 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:48.730232 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.730914 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.732271 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.732971 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733094 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.733243 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.733284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":63,"threads_started":1}
I20260619 20:58:48.733701 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35557 (local address 127.0.0.1:36546)
0619 20:58:48.729494 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.729752 (+ 258us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.729842 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.730042 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.730042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.730045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:48.730442 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.730445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.731660 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.731660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.732562 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.732564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.733465 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.733534 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733537 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.733597 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.733631 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":53,"threads_started":1}
I20260619 20:58:48.733744 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.98.2:35557)
0619 20:58:48.729631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.729635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.729808 (+ 173us) server_negotiation.cc:407] Connection header received
0619 20:58:48.729930 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.729933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.729948 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.730042 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:48.730544 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.731571 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.732641 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.733315 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733373 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733429 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.733623 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.733658 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260619 20:58:48.734030 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39363 (local address 127.0.0.1:43312)
0619 20:58:48.729688 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.729699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.729770 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.730031 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.730033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.730036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:48.730094 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.730097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.731402 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.731403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.732410 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.732412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.733146 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.733220 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733225 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.733276 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.733317 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":34,"threads_started":1}
I20260619 20:58:48.734196 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.729608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.98.3:39363)
0619 20:58:48.729764 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.729768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.729779 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:58:48.729841 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.729843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.729860 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.729936 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:48.730159 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.731297 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.732483 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.732994 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.733083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.733137 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.733376 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.734129 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260619 20:58:48.734120 17508 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce (DEFAULT_TABLE table=test-table [id=6e953bc95f9d4894a6e99bd2627adf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:48.734486 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:48.734382 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce (DEFAULT_TABLE table=test-table [id=6e953bc95f9d4894a6e99bd2627adf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:48.734719 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:48.737030 17856 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: Bootstrap starting.
I20260619 20:58:48.737282 17857 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Bootstrap starting.
I20260619 20:58:48.737624 17776 tablet_service.cc:1467] Processing CreateTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce (DEFAULT_TABLE table=test-table [id=6e953bc95f9d4894a6e99bd2627adf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:48.737686 17856 tablet_bootstrap.cc:654] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:48.737845 17857 tablet_bootstrap.cc:654] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:48.737931 17776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:48.738356 17856 log.cc:824] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:48.738440 17857 log.cc:824] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:48.739094 17857 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: No bootstrap required, opened a new log
I20260619 20:58:48.739143 17857 ts_tablet_manager.cc:1397] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:58:48.740537 17857 raft_consensus.cc:348] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.740741 17857 raft_consensus.cc:374] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:48.740794 17857 raft_consensus.cc:729] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30c25ca256449e8498a3f87f60427b, State: Initialized, Role: FOLLOWER
I20260619 20:58:48.740833 17860 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Bootstrap starting.
I20260619 20:58:48.740918 17857 consensus_queue.cc:260] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.741221 17857 ts_tablet_manager.cc:1428] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:58:48.741375 17708 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
I20260619 20:58:48.741516 17860 tablet_bootstrap.cc:654] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:48.741897 17856 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: No bootstrap required, opened a new log
I20260619 20:58:48.741962 17856 ts_tablet_manager.cc:1397] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 20:58:48.742646 17860 log.cc:824] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:48.743450 17860 tablet_bootstrap.cc:492] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: No bootstrap required, opened a new log
I20260619 20:58:48.743516 17860 ts_tablet_manager.cc:1397] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:58:48.744055 17856 raft_consensus.cc:348] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.744199 17856 raft_consensus.cc:374] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:48.744252 17856 raft_consensus.cc:729] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f4ac83a2b74785a0c60e20f06774ad, State: Initialized, Role: FOLLOWER
I20260619 20:58:48.744390 17856 consensus_queue.cc:260] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.744643 17856 ts_tablet_manager.cc:1428] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:48.744666 17574 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
I20260619 20:58:48.745538 17860 raft_consensus.cc:348] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.745687 17860 raft_consensus.cc:374] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:48.745720 17860 raft_consensus.cc:729] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3fd0f2e5384f138005e5b71174484a, State: Initialized, Role: FOLLOWER
I20260619 20:58:48.745824 17860 consensus_queue.cc:260] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.746093 17860 ts_tablet_manager.cc:1428] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:48.746093 17842 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
W20260619 20:58:48.790475 17709 tablet.cc:2369] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:48.819870 17861 raft_consensus.cc:482] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:48.819986 17861 raft_consensus.cc:504] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.820381 17861 leader_election.cc:288] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0f4ac83a2b74785a0c60e20f06774ad (127.0.98.1:44265), 0a3fd0f2e5384f138005e5b71174484a (127.0.98.3:39363)
I20260619 20:58:48.823208 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.820586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44265 (local address 127.0.98.2:49449)
0619 20:58:48.820667 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.820681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.820742 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.820905 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.820905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.820907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:48.820984 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.820986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.821609 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.821610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.822371 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.822373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.822918 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.822919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.823002 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.823004 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.823082 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.823137 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":17}
I20260619 20:58:48.823210 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.820641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49449 (local address 127.0.98.1:44265)
0619 20:58:48.820690 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.820692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.820791 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:58:48.820822 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.820823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.820836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.820899 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:48.821058 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.821526 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.822463 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.822832 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.822924 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.822934 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.823111 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.823150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:58:48.823350 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.820790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59463 (local address 127.0.98.3:39363)
0619 20:58:48.820843 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.820845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.820941 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:58:48.821023 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.821024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.821032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.821080 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:48.821225 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.821676 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.822624 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.822985 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.823063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.823071 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.823240 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.823279 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:58:48.823359 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.820700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39363 (local address 127.0.98.2:59463)
0619 20:58:48.820874 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.820883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:48.820954 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:48.821083 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:48.821083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:48.821084 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:48.821155 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.821157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.821764 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.821765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.822544 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:48.822546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.823065 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.823066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:48.823152 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.823155 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:48.823233 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:58:48.823269 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
I20260619 20:58:48.823606 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5e9853bbdc46e0b86fe0e6fa64e8ce" candidate_uuid: "4d30c25ca256449e8498a3f87f60427b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" is_pre_election: true
I20260619 20:58:48.823766 17527 raft_consensus.cc:2454] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30c25ca256449e8498a3f87f60427b in term 0.
I20260619 20:58:48.823730 17796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5e9853bbdc46e0b86fe0e6fa64e8ce" candidate_uuid: "4d30c25ca256449e8498a3f87f60427b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3fd0f2e5384f138005e5b71174484a" is_pre_election: true
I20260619 20:58:48.823866 17796 raft_consensus.cc:2454] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30c25ca256449e8498a3f87f60427b in term 0.
I20260619 20:58:48.824023 17596 leader_election.cc:302] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d30c25ca256449e8498a3f87f60427b, d0f4ac83a2b74785a0c60e20f06774ad; no voters:
I20260619 20:58:48.824152 17861 raft_consensus.cc:2781] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:48.824208 17861 raft_consensus.cc:482] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:48.824235 17861 raft_consensus.cc:3037] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:48.824885 17861 raft_consensus.cc:504] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.825006 17861 leader_election.cc:288] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [CANDIDATE]: Term 1 election: Requested vote from peers d0f4ac83a2b74785a0c60e20f06774ad (127.0.98.1:44265), 0a3fd0f2e5384f138005e5b71174484a (127.0.98.3:39363)
I20260619 20:58:48.825169 17527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5e9853bbdc46e0b86fe0e6fa64e8ce" candidate_uuid: "4d30c25ca256449e8498a3f87f60427b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f4ac83a2b74785a0c60e20f06774ad"
I20260619 20:58:48.825212 17796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b5e9853bbdc46e0b86fe0e6fa64e8ce" candidate_uuid: "4d30c25ca256449e8498a3f87f60427b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3fd0f2e5384f138005e5b71174484a"
I20260619 20:58:48.825269 17796 raft_consensus.cc:3037] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:48.825273 17527 raft_consensus.cc:3037] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:48.825941 17796 raft_consensus.cc:2454] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30c25ca256449e8498a3f87f60427b in term 1.
I20260619 20:58:48.826090 17527 raft_consensus.cc:2454] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30c25ca256449e8498a3f87f60427b in term 1.
I20260619 20:58:48.826110 17596 leader_election.cc:302] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3fd0f2e5384f138005e5b71174484a, 4d30c25ca256449e8498a3f87f60427b; no voters:
I20260619 20:58:48.826288 17861 raft_consensus.cc:2781] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:48.826483 17861 raft_consensus.cc:686] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 1 LEADER]: Becoming Leader. State: Replica: 4d30c25ca256449e8498a3f87f60427b, State: Running, Role: LEADER
I20260619 20:58:48.826647 17861 consensus_queue.cc:237] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } }
I20260619 20:58:48.827473 17381 catalog_manager.cc:5495] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d30c25ca256449e8498a3f87f60427b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d30c25ca256449e8498a3f87f60427b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30c25ca256449e8498a3f87f60427b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 } health_report { overall_health: UNKNOWN } } }
I20260619 20:58:48.847798 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.845595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.0.98.2:35557)
0619 20:58:48.845649 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:48.845651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:48.845662 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:58:48.845695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:48.845695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:48.845722 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:48.845781 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:48.845898 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.846405 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.847104 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:48.847281 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:48.847329 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:48.847476 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:48.847566 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:48.847622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:48.847657 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:58:48.847673 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:58:48.849498 17527 raft_consensus.cc:1261] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P d0f4ac83a2b74785a0c60e20f06774ad [term 1 FOLLOWER]: Refusing update from remote peer 4d30c25ca256449e8498a3f87f60427b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:48.849498 17796 raft_consensus.cc:1261] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 1 FOLLOWER]: Refusing update from remote peer 4d30c25ca256449e8498a3f87f60427b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:48.849867 17861 consensus_queue.cc:1035] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3fd0f2e5384f138005e5b71174484a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:48.850023 17866 consensus_queue.cc:1035] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f4ac83a2b74785a0c60e20f06774ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:48.852274 17869 mvcc.cc:205] Tried to move back new op lower bound from 7298673577364938752 to 7298673577274540032. Current Snapshot: MvccSnapshot[applied={T|T < 7298673577364938752}]
I20260619 20:58:48.854326 17381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:48.854434 17381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:48.855152 17381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32742dfa9c4e46178ef1c26f6d0b8758: Sending DeleteTablet for 3 replicas of tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce
I20260619 20:58:48.855500 17642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36546
I20260619 20:58:48.855597 17776 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43312
I20260619 20:58:48.855799 17879 tablet_replica.cc:330] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: stopping tablet replica
I20260619 20:58:48.855798 17508 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46068
I20260619 20:58:48.855893 17879 raft_consensus.cc:2229] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:48.855993 17879 raft_consensus.cc:2258] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:48.856207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17445
I20260619 20:58:48.857334 17879 ts_tablet_manager.cc:1905] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:48.859382 17879 ts_tablet_manager.cc:1918] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:48.859457 17879 log.cc:1196] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/8b5e9853bbdc46e0b86fe0e6fa64e8ce
I20260619 20:58:48.859736 17879 ts_tablet_manager.cc:1939] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 0a3fd0f2e5384f138005e5b71174484a: Deleting consensus metadata
I20260619 20:58:48.860136 17368 catalog_manager.cc:4841] TS 0a3fd0f2e5384f138005e5b71174484a (127.0.98.3:39363): tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce (table test-table [id=6e953bc95f9d4894a6e99bd2627adf05]) successfully deleted
I20260619 20:58:48.862327 17878 tablet_replica.cc:330] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: stopping tablet replica
I20260619 20:58:48.862432 17878 raft_consensus.cc:2229] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:48.862561 17878 raft_consensus.cc:2258] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:48.863013 17878 ts_tablet_manager.cc:1905] T 8b5e9853bbdc46e0b86fe0e6fa64e8ce P 4d30c25ca256449e8498a3f87f60427b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:58:48.863157 17368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44265 (error 108)
I20260619 20:58:48.863407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17579
I20260619 20:58:48.863696 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.863518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44265 (local address 127.0.0.1:46084)
0619 20:58:48.863610 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.863641 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 20:58:48.866329 17368 catalog_manager.cc:4584] TS d0f4ac83a2b74785a0c60e20f06774ad (127.0.98.1:44265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b5e9853bbdc46e0b86fe0e6fa64e8ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44265: connect: Connection refused (error 111)
I20260619 20:58:48.868149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17713
I20260619 20:58:48.868263 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:48.868123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35557 (local address 127.0.0.1:36570)
0619 20:58:48.868199 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:48.868230 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:58:48.872422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17351
2026-06-19T20:58:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1930 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T20:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:48Z Disabled control of system clock
I20260619 20:58:48.899158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36497
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:48.977790 17885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:48.978011 17885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:48.978031 17885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:48.979197 17885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:48.979238 17885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:48.979267 17885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:48.979316 17885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:48.979333 17885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:48.980643 17885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:48.980907 17885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:48.981181 17885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:48.985733 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.985741 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:48.985813 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:48.985895 17885 server_base.cc:1034] running on GCE node
I20260619 20:58:48.986343 17885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:48.986686 17885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:48.987840 17885 hybrid_clock.cc:648] HybridClock initialized: now 1781902728987819 us; error 35 us; skew 500 ppm
I20260619 20:58:48.989092 17885 webserver.cc:460] Webserver started at http://127.0.98.62:45385/ using document root <none> and password file <none>
I20260619 20:58:48.989398 17885 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:48.989445 17885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:48.989632 17885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:48.990676 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "b0a2c0891ef14435a25fe967b5117228"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "763aed4cc7e2c55065c60c5b8a36b69a"
server_key_iv: "65a4c918185a3713f5c8689ae3b2cd08"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.991020 17885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0a2c0891ef14435a25fe967b5117228"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "763aed4cc7e2c55065c60c5b8a36b69a"
server_key_iv: "65a4c918185a3713f5c8689ae3b2cd08"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.992245 17885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:48.992961 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:48.993158 17885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:48.993221 17885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "b0a2c0891ef14435a25fe967b5117228"
format_stamp: "Formatted at 2026-06-19 20:58:48 on dist-test-slave-8bgz"
server_key: "763aed4cc7e2c55065c60c5b8a36b69a"
server_key_iv: "65a4c918185a3713f5c8689ae3b2cd08"
server_key_version: "encryptionkey@0"
I20260619 20:58:48.993314 17885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:49.022385 17885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:49.022706 17885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:49.022852 17885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:49.026767 17885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36497
I20260619 20:58:49.026803 17951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36497 every 8 connection(s)
I20260619 20:58:49.027145 17885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:49.028061 17952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:49.030295 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: Bootstrap starting.
I20260619 20:58:49.030851 17952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:49.031136 17952 log.cc:824] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:49.031791 17952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: No bootstrap required, opened a new log
I20260619 20:58:49.033448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17885
I20260619 20:58:49.033363 17952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260619 20:58:49.033542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:49.033530 17952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:49.033567 17952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a2c0891ef14435a25fe967b5117228, State: Initialized, Role: FOLLOWER
I20260619 20:58:49.033725 392 external_mini_cluster.cc:1428] Setting key 5c10c766edc8ef7a4fec2671a01c9cb0
I20260619 20:58:49.033725 17952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260619 20:58:49.033799 17952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:49.033833 17952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:49.033881 17952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:49.034529 17952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260619 20:58:49.034611 17952 leader_election.cc:302] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0a2c0891ef14435a25fe967b5117228; no voters:
I20260619 20:58:49.034806 17952 leader_election.cc:288] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:49.034829 17957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:49.034945 17957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [term 1 LEADER]: Becoming Leader. State: Replica: b0a2c0891ef14435a25fe967b5117228, State: Running, Role: LEADER
I20260619 20:58:49.035022 17957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } }
I20260619 20:58:49.035061 17952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:49.035425 17958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0a2c0891ef14435a25fe967b5117228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } } }
I20260619 20:58:49.035522 17958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:49.035696 17959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a2c0891ef14435a25fe967b5117228. Latest consensus state: current_term: 1 leader_uuid: "b0a2c0891ef14435a25fe967b5117228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a2c0891ef14435a25fe967b5117228" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36497 } } }
I20260619 20:58:49.035755 17959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:49.035847 17965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:49.036275 17965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:49.037956 17965 catalog_manager.cc:1347] Generated new cluster ID: 3dd33061952f47bd96568a4bb5954ab1
I20260619 20:58:49.038008 17965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:49.039613 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.034262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.98.62:36497)
0619 20:58:49.034522 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.034529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.034558 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:58:49.034681 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.034688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.034752 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.034897 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:49.035324 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.036671 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.037606 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.038185 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.038309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.038968 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:49.038971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:49.038972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:49.039165 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:49.039199 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:49.039211 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:49.039275 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.039353 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.039423 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":113,"threads_started":1}
I20260619 20:58:49.052647 17965 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:49.053138 17965 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:49.057020 17965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: Generated new TSK 0
I20260619 20:58:49.057192 17965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:49.059387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--builtin_ntp_servers=127.0.98.20:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:49.138820 17976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:49.139043 17976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:49.139088 17976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:49.140218 17976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:49.140278 17976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:49.140321 17976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:49.141681 17976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:49.141948 17976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:49.142292 17976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:49.147033 17981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.147275 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.147293 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:49.147630 17976 server_base.cc:1034] running on GCE node
I20260619 20:58:49.147841 17976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:49.148113 17976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:49.149291 17976 hybrid_clock.cc:648] HybridClock initialized: now 1781902729149278 us; error 34 us; skew 500 ppm
I20260619 20:58:49.150630 17976 webserver.cc:460] Webserver started at http://127.0.98.1:44163/ using document root <none> and password file <none>
I20260619 20:58:49.150882 17976 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:49.150947 17976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:49.151098 17976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:49.152140 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fb947aaca184479ac260d260e77dbf7"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "47bd0a44269c6f99bcea03ea41465da8"
server_key_iv: "d2bc59f28543dc89ff8808ac3b99ebf3"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.152490 17976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fb947aaca184479ac260d260e77dbf7"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "47bd0a44269c6f99bcea03ea41465da8"
server_key_iv: "d2bc59f28543dc89ff8808ac3b99ebf3"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.153882 17976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:58:49.154739 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.154909 17976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:58:49.155002 17976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "8fb947aaca184479ac260d260e77dbf7"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "47bd0a44269c6f99bcea03ea41465da8"
server_key_iv: "d2bc59f28543dc89ff8808ac3b99ebf3"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.155083 17976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:49.169497 17976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:49.169803 17976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:49.169950 17976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:49.170166 17976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:49.170507 17976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:49.170568 17976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.170608 17976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:49.170637 17976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.176110 17976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33113
I20260619 20:58:49.176175 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33113 every 8 connection(s)
I20260619 20:58:49.176488 17976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:49.181284 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.177214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54521 (local address 127.0.98.62:36497)
0619 20:58:49.177258 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.177260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.177485 (+ 225us) server_negotiation.cc:407] Connection header received
0619 20:58:49.177918 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.177918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.177927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.177987 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:49.178663 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.179157 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.180065 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.180245 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.180298 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.181055 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:49.181056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:49.181057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:49.181061 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:49.181075 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:49.181079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:49.181123 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.181201 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.181227 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:58:49.181480 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.177172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.1:54521)
0619 20:58:49.177350 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.177388 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.177842 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.178036 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.178039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.178046 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:49.178569 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.178575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.179239 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.179240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.179983 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.179984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.180316 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.180317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.180417 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.180921 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:49.180922 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:49.180946 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:49.180966 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:49.181133 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:49.181136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:49.181138 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.181188 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.181264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260619 20:58:49.181990 18104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260619 20:58:49.182085 18104 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:49.182307 18104 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
I20260619 20:58:49.182796 17916 ts_manager.cc:194] Registered new tserver with Master: 8fb947aaca184479ac260d260e77dbf7 (127.0.98.1:33113)
I20260619 20:58:49.183378 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54521
I20260619 20:58:49.184173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 17976
I20260619 20:58:49.184280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:49.184486 392 external_mini_cluster.cc:1428] Setting key 6d97206e0cb645b396c029c06b6c7782
I20260619 20:58:49.185339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--builtin_ntp_servers=127.0.98.20:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:49.262988 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:49.263173 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:49.263199 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:49.264323 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:49.264365 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:49.264412 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:49.265712 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:49.265940 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:49.266258 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:49.271147 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.271174 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.271174 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:49.271337 18107 server_base.cc:1034] running on GCE node
I20260619 20:58:49.271562 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:49.271742 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:49.272900 18107 hybrid_clock.cc:648] HybridClock initialized: now 1781902729272876 us; error 34 us; skew 500 ppm
I20260619 20:58:49.274170 18107 webserver.cc:460] Webserver started at http://127.0.98.2:35249/ using document root <none> and password file <none>
I20260619 20:58:49.274427 18107 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:49.274495 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:49.274624 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:49.275588 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "8bcd4d1e28024352b3e4c32c35bd4f86"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "11abb2a1a57b59230e7c2c77bdf7c5a9"
server_key_iv: "076d94dea08734d2ec5f301e85bd3e3f"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.275928 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8bcd4d1e28024352b3e4c32c35bd4f86"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "11abb2a1a57b59230e7c2c77bdf7c5a9"
server_key_iv: "076d94dea08734d2ec5f301e85bd3e3f"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.277174 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:49.277900 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.278074 18107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:49.278167 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "8bcd4d1e28024352b3e4c32c35bd4f86"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "11abb2a1a57b59230e7c2c77bdf7c5a9"
server_key_iv: "076d94dea08734d2ec5f301e85bd3e3f"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.278260 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:49.297847 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:49.298131 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:49.298305 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:49.298511 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:49.298823 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:49.298877 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.298918 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:49.298944 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.304656 18107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37131
I20260619 20:58:49.304709 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37131 every 8 connection(s)
I20260619 20:58:49.304994 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:49.309641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18107
I20260619 20:58:49.309733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:49.309767 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.305652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45049 (local address 127.0.98.62:36497)
0619 20:58:49.305698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.305700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.305895 (+ 195us) server_negotiation.cc:407] Connection header received
0619 20:58:49.306347 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.306348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.306359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.306421 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:49.307001 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.307464 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.308377 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.308563 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.308620 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.309415 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:49.309416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:49.309417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:49.309421 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:49.309455 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:49.309460 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:49.309520 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.309622 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.309654 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:58:49.309921 392 external_mini_cluster.cc:1428] Setting key 3b81988b8f5173092456065d97ddef83
I20260619 20:58:49.309937 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.305607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.2:45049)
0619 20:58:49.305771 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.305810 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.306269 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.306472 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.306475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.306482 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:49.306937 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.306940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.307537 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.307537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.308299 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.308300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.308642 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.308642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.308739 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.309280 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:49.309281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:49.309305 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:49.309327 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:49.309528 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:49.309534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:49.309537 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.309615 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.309723 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":89,"threads_started":1}
I20260619 20:58:49.310479 18235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260619 20:58:49.310598 18235 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:49.310809 18235 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
I20260619 20:58:49.310942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--builtin_ntp_servers=127.0.98.20:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:58:49.311198 17916 ts_manager.cc:194] Registered new tserver with Master: 8bcd4d1e28024352b3e4c32c35bd4f86 (127.0.98.2:37131)
I20260619 20:58:49.311564 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45049
W20260619 20:58:49.392236 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:49.392426 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:49.392449 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:49.393594 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:49.393637 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:49.393687 18238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:49.395035 18238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:49.395305 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:49.395558 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:49.400369 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.400405 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:49.400345 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:49.400499 18238 server_base.cc:1034] running on GCE node
I20260619 20:58:49.400734 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:49.400970 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:49.402128 18238 hybrid_clock.cc:648] HybridClock initialized: now 1781902729402115 us; error 27 us; skew 500 ppm
I20260619 20:58:49.403314 18238 webserver.cc:460] Webserver started at http://127.0.98.3:44853/ using document root <none> and password file <none>
I20260619 20:58:49.403556 18238 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:49.403623 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:49.403774 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:49.404768 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "0183a185d115432c8524b96131e6226c"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "94e403093f7e285f20e3575071862558"
server_key_iv: "e165e0c6549face048fef2e6b8d48fda"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.405110 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0183a185d115432c8524b96131e6226c"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "94e403093f7e285f20e3575071862558"
server_key_iv: "e165e0c6549face048fef2e6b8d48fda"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.406394 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:49.407150 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.407325 18238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:49.407413 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "0183a185d115432c8524b96131e6226c"
format_stamp: "Formatted at 2026-06-19 20:58:49 on dist-test-slave-8bgz"
server_key: "94e403093f7e285f20e3575071862558"
server_key_iv: "e165e0c6549face048fef2e6b8d48fda"
server_key_version: "encryptionkey@0"
I20260619 20:58:49.407493 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:49.427124 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:49.427399 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:49.427541 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:49.427757 18238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:49.428058 18238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:49.428110 18238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.428151 18238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:49.428179 18238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:49.433557 18238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36597
I20260619 20:58:49.433632 18365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36597 every 8 connection(s)
I20260619 20:58:49.433950 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:49.435760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18238
I20260619 20:58:49.435859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:49.436090 392 external_mini_cluster.cc:1428] Setting key bece2923155402750ac97d7a5bac0f72
I20260619 20:58:49.439350 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.434641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34789 (local address 127.0.98.62:36497)
0619 20:58:49.434687 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.434689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.434889 (+ 200us) server_negotiation.cc:407] Connection header received
0619 20:58:49.435403 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.435404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.435417 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.435466 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:49.436341 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.436907 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.437874 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.438123 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.438200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.439087 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:49.439089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:49.439089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:49.439093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:49.439108 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:49.439112 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:49.439155 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.439234 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.439273 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:58:49.439555 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.434607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36497 (local address 127.0.98.3:34789)
0619 20:58:49.434772 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.434802 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.435303 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.435551 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.435555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.435582 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:49.436251 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.436257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.436997 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.436998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.437785 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.437787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.438226 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.438228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.438376 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.438936 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:49.438936 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:49.438963 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:49.438984 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:49.439166 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:49.439170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:49.439173 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.439227 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.439335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260619 20:58:49.440023 18366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36497
I20260619 20:58:49.440094 18366 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:49.440274 18366 heartbeater.cc:511] Master 127.0.98.62:36497 requested a full tablet report, sending...
I20260619 20:58:49.440634 17916 ts_manager.cc:194] Registered new tserver with Master: 0183a185d115432c8524b96131e6226c (127.0.98.3:36597)
I20260619 20:58:49.440982 17916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34789
I20260619 20:58:49.447638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:49.451759 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.448961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.98.62:36497)
0619 20:58:49.449003 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.449005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.449075 (+ 70us) server_negotiation.cc:407] Connection header received
0619 20:58:49.449145 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.449146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.449156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.449219 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:49.449396 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.449844 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.450651 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.450838 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.450917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.451533 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:49.451534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:49.451534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:49.451538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:49.451550 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:49.451554 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:49.451604 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.451666 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.451686 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:58:49.453265 17916 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:49.454874 17916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:49.460999 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.456997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37131 (local address 127.0.0.1:35680)
0619 20:58:49.457226 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.457236 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.457277 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.457708 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.457709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.457711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:49.457742 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.457742 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.458784 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.458785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.460034 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.460036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.460725 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.460799 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.460813 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.460882 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.460930 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260619 20:58:49.461009 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.457236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.0.98.2:37131)
0619 20:58:49.457424 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.457427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.457435 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:58:49.457464 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.457465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.457476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.457565 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:49.457819 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.458710 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460134 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.460621 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460706 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.460793 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.460895 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.460932 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260619 20:58:49.461143 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.456791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.0.1:57908)
0619 20:58:49.457117 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.457157 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.457258 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.457641 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.457641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.457643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:49.457691 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.457693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.459025 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.459026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.459919 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.459922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.460889 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.460946 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.460950 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.460998 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.461035 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":64,"threads_started":1}
I20260619 20:58:49.461413 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.456881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.98.3:36597)
0619 20:58:49.457042 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.457047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.457224 (+ 177us) server_negotiation.cc:407] Connection header received
0619 20:58:49.457351 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.457354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.457368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.457421 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:49.457776 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.458928 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460001 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.460702 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.460853 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.461303 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.461340 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":103,"threads_started":1}
I20260619 20:58:49.461575 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.456822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33113 (local address 127.0.0.1:45026)
0619 20:58:49.457126 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.457153 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.457228 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.457575 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.457578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.457580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:49.457634 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.457635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.459007 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.459008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.460078 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.460079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.461004 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.461005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.461390 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.461394 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.461457 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.461496 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260619 20:58:49.461784 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.456938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.1:33113)
0619 20:58:49.457406 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.457410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.457420 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:49.457454 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.457456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.457470 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.457572 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:49.457698 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.458924 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460153 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.460841 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.460900 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.460974 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.461662 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.461709 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":78,"threads_started":1}
I20260619 20:58:49.461673 18169 tablet_service.cc:1467] Processing CreateTablet for tablet a350aa5fb9864df8baf64d2775c2da24 (DEFAULT_TABLE table=test-table [id=710490bcd7454ce29741f85b3c889b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:49.462013 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a350aa5fb9864df8baf64d2775c2da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:49.462083 18300 tablet_service.cc:1467] Processing CreateTablet for tablet a350aa5fb9864df8baf64d2775c2da24 (DEFAULT_TABLE table=test-table [id=710490bcd7454ce29741f85b3c889b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:49.462426 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a350aa5fb9864df8baf64d2775c2da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:49.462855 18038 tablet_service.cc:1467] Processing CreateTablet for tablet a350aa5fb9864df8baf64d2775c2da24 (DEFAULT_TABLE table=test-table [id=710490bcd7454ce29741f85b3c889b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:49.463162 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a350aa5fb9864df8baf64d2775c2da24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:49.464636 18380 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Bootstrap starting.
I20260619 20:58:49.465188 18380 tablet_bootstrap.cc:654] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:49.465516 18380 log.cc:824] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:49.465620 18381 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Bootstrap starting.
I20260619 20:58:49.466163 18381 tablet_bootstrap.cc:654] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:49.466238 18383 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: Bootstrap starting.
I20260619 20:58:49.466518 18381 log.cc:824] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:49.466851 18383 tablet_bootstrap.cc:654] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:49.467016 18380 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: No bootstrap required, opened a new log
I20260619 20:58:49.467092 18380 ts_tablet_manager.cc:1397] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:58:49.467180 18383 log.cc:824] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:49.467933 18383 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: No bootstrap required, opened a new log
I20260619 20:58:49.467986 18383 ts_tablet_manager.cc:1397] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:49.468667 18380 raft_consensus.cc:348] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.468812 18380 raft_consensus.cc:374] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:49.468858 18380 raft_consensus.cc:729] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcd4d1e28024352b3e4c32c35bd4f86, State: Initialized, Role: FOLLOWER
I20260619 20:58:49.468972 18380 consensus_queue.cc:260] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.469220 18380 ts_tablet_manager.cc:1428] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:58:49.469250 18235 heartbeater.cc:503] Master 127.0.98.62:36497 was elected leader, sending a full tablet report...
I20260619 20:58:49.469854 18381 tablet_bootstrap.cc:492] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: No bootstrap required, opened a new log
I20260619 20:58:49.469920 18381 ts_tablet_manager.cc:1397] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:58:49.470109 18383 raft_consensus.cc:348] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.470299 18383 raft_consensus.cc:374] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:49.470336 18383 raft_consensus.cc:729] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb947aaca184479ac260d260e77dbf7, State: Initialized, Role: FOLLOWER
I20260619 20:58:49.470616 18383 consensus_queue.cc:260] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.470913 18383 ts_tablet_manager.cc:1428] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:49.470953 18104 heartbeater.cc:503] Master 127.0.98.62:36497 was elected leader, sending a full tablet report...
I20260619 20:58:49.472347 18381 raft_consensus.cc:348] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.472445 18381 raft_consensus.cc:374] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:49.472463 18381 raft_consensus.cc:729] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0183a185d115432c8524b96131e6226c, State: Initialized, Role: FOLLOWER
I20260619 20:58:49.472534 18381 consensus_queue.cc:260] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.472708 18381 ts_tablet_manager.cc:1428] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:49.472793 18366 heartbeater.cc:503] Master 127.0.98.62:36497 was elected leader, sending a full tablet report...
I20260619 20:58:49.536358 18386 raft_consensus.cc:482] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:49.536670 18386 raft_consensus.cc:504] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.537052 18386 leader_election.cc:288] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0183a185d115432c8524b96131e6226c (127.0.98.3:36597), 8fb947aaca184479ac260d260e77dbf7 (127.0.98.1:33113)
I20260619 20:58:49.540934 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.537323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40709 (local address 127.0.98.1:33113)
0619 20:58:49.537364 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.537366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.537467 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:58:49.537497 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.537498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.537509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.537563 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:49.537710 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.538481 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.539998 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.540542 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.540616 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.540625 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.540826 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.540869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:58:49.540934 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.537253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37101 (local address 127.0.98.3:36597)
0619 20:58:49.537303 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.537306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.537316 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:49.537393 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.537394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.537408 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.537485 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:49.537692 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.538479 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.539998 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.540542 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.540616 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.540625 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.540826 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.540867 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:58:49.540946 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.537204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36597 (local address 127.0.98.2:37101)
0619 20:58:49.537260 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.537271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.537314 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.537506 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.537507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.537508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:49.537598 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.537600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.538557 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.538558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.539908 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.539911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.540626 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.540627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.540746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.540751 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.540808 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.540852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":76}
I20260619 20:58:49.540951 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.537221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33113 (local address 127.0.98.2:40709)
0619 20:58:49.537381 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.537390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:49.537443 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:49.537574 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:49.537574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:49.537576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:49.537643 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.537644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.538557 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.538558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.539911 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:49.539912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.540626 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.540627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:49.540750 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.540752 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:49.540808 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:58:49.540851 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":137,"thread_start_us":85,"threads_started":1}
I20260619 20:58:49.541311 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a350aa5fb9864df8baf64d2775c2da24" candidate_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0183a185d115432c8524b96131e6226c" is_pre_election: true
I20260619 20:58:49.541466 18320 raft_consensus.cc:2454] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcd4d1e28024352b3e4c32c35bd4f86 in term 0.
I20260619 20:58:49.541591 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a350aa5fb9864df8baf64d2775c2da24" candidate_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb947aaca184479ac260d260e77dbf7" is_pre_election: true
I20260619 20:58:49.541704 18125 leader_election.cc:302] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0183a185d115432c8524b96131e6226c, 8bcd4d1e28024352b3e4c32c35bd4f86; no voters:
I20260619 20:58:49.541735 18058 raft_consensus.cc:2454] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcd4d1e28024352b3e4c32c35bd4f86 in term 0.
I20260619 20:58:49.541888 18386 raft_consensus.cc:2781] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:49.541965 18386 raft_consensus.cc:482] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:49.541986 18386 raft_consensus.cc:3037] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:49.542662 18386 raft_consensus.cc:504] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.542799 18386 leader_election.cc:288] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [CANDIDATE]: Term 1 election: Requested vote from peers 0183a185d115432c8524b96131e6226c (127.0.98.3:36597), 8fb947aaca184479ac260d260e77dbf7 (127.0.98.1:33113)
I20260619 20:58:49.542940 18320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a350aa5fb9864df8baf64d2775c2da24" candidate_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0183a185d115432c8524b96131e6226c"
I20260619 20:58:49.542958 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a350aa5fb9864df8baf64d2775c2da24" candidate_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb947aaca184479ac260d260e77dbf7"
I20260619 20:58:49.543036 18320 raft_consensus.cc:3037] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:49.543033 18058 raft_consensus.cc:3037] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:49.543737 18058 raft_consensus.cc:2454] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcd4d1e28024352b3e4c32c35bd4f86 in term 1.
I20260619 20:58:49.543848 18320 raft_consensus.cc:2454] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcd4d1e28024352b3e4c32c35bd4f86 in term 1.
I20260619 20:58:49.543910 18122 leader_election.cc:302] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bcd4d1e28024352b3e4c32c35bd4f86, 8fb947aaca184479ac260d260e77dbf7; no voters:
I20260619 20:58:49.544046 18386 raft_consensus.cc:2781] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:49.544191 18386 raft_consensus.cc:686] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 1 LEADER]: Becoming Leader. State: Replica: 8bcd4d1e28024352b3e4c32c35bd4f86, State: Running, Role: LEADER
I20260619 20:58:49.544281 18386 consensus_queue.cc:237] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } }
I20260619 20:58:49.545004 17916 catalog_manager.cc:5495] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bcd4d1e28024352b3e4c32c35bd4f86 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bcd4d1e28024352b3e4c32c35bd4f86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37131 } health_report { overall_health: HEALTHY } } }
I20260619 20:58:49.553074 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.550713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.98.2:37131)
0619 20:58:49.550754 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:49.550756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:49.550765 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:58:49.550829 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:49.550829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:49.550844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:49.550911 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:49.551010 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.551473 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.552301 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:49.552501 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:49.552580 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:49.552748 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:49.552840 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:49.552922 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:49.552973 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:58:49.552989 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:58:49.554944 18320 raft_consensus.cc:1261] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 1 FOLLOWER]: Refusing update from remote peer 8bcd4d1e28024352b3e4c32c35bd4f86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:49.554976 18058 raft_consensus.cc:1261] T a350aa5fb9864df8baf64d2775c2da24 P 8fb947aaca184479ac260d260e77dbf7 [term 1 FOLLOWER]: Refusing update from remote peer 8bcd4d1e28024352b3e4c32c35bd4f86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:49.555271 18390 consensus_queue.cc:1035] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb947aaca184479ac260d260e77dbf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:58:49.555464 18236 tablet.cc:2369] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:49.555614 18386 consensus_queue.cc:1035] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0183a185d115432c8524b96131e6226c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:49.558841 18393 mvcc.cc:205] Tried to move back new op lower bound from 7298673580254748672 to 7298673580213927936. Current Snapshot: MvccSnapshot[applied={T|T < 7298673580254748672 or (T in {7298673580254748672})}]
I20260619 20:58:49.558962 18394 mvcc.cc:205] Tried to move back new op lower bound from 7298673580254748672 to 7298673580213927936. Current Snapshot: MvccSnapshot[applied={T|T < 7298673580254748672}]
I20260619 20:58:49.559625 18395 mvcc.cc:205] Tried to move back new op lower bound from 7298673580254748672 to 7298673580213927936. Current Snapshot: MvccSnapshot[applied={T|T < 7298673580254748672 or (T in {7298673580254748672})}]
I20260619 20:58:49.559768 17915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:49.559857 17915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:49.560578 17915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0a2c0891ef14435a25fe967b5117228: Sending DeleteTablet for 3 replicas of tablet a350aa5fb9864df8baf64d2775c2da24
I20260619 20:58:49.560885 18038 tablet_service.cc:1514] Processing DeleteTablet for tablet a350aa5fb9864df8baf64d2775c2da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:49 UTC) from {username='slave'} at 127.0.0.1:45026
I20260619 20:58:49.560912 18300 tablet_service.cc:1514] Processing DeleteTablet for tablet a350aa5fb9864df8baf64d2775c2da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:49 UTC) from {username='slave'} at 127.0.0.1:57908
I20260619 20:58:49.560912 18169 tablet_service.cc:1514] Processing DeleteTablet for tablet a350aa5fb9864df8baf64d2775c2da24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:49 UTC) from {username='slave'} at 127.0.0.1:35680
I20260619 20:58:49.561095 18402 tablet_replica.cc:330] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: stopping tablet replica
I20260619 20:58:49.561184 18402 raft_consensus.cc:2229] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:49.561287 18402 raft_consensus.cc:2258] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:49.561378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17976
I20260619 20:58:49.561301 18403 tablet_replica.cc:330] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: stopping tablet replica
I20260619 20:58:49.562598 18403 raft_consensus.cc:2229] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:49.562711 18403 raft_consensus.cc:2258] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:49.563077 18403 ts_tablet_manager.cc:1905] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:49.564342 18402 ts_tablet_manager.cc:1905] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:49.564945 18403 ts_tablet_manager.cc:1918] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:49.565070 18403 log.cc:1196] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/a350aa5fb9864df8baf64d2775c2da24
I20260619 20:58:49.565376 18403 ts_tablet_manager.cc:1939] T a350aa5fb9864df8baf64d2775c2da24 P 0183a185d115432c8524b96131e6226c: Deleting consensus metadata
I20260619 20:58:49.565805 17903 catalog_manager.cc:4841] TS 0183a185d115432c8524b96131e6226c (127.0.98.3:36597): tablet a350aa5fb9864df8baf64d2775c2da24 (table test-table [id=710490bcd7454ce29741f85b3c889b02]) successfully deleted
I20260619 20:58:49.565946 18402 ts_tablet_manager.cc:1918] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:49.566010 18402 log.cc:1196] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/a350aa5fb9864df8baf64d2775c2da24
I20260619 20:58:49.566291 18402 ts_tablet_manager.cc:1939] T a350aa5fb9864df8baf64d2775c2da24 P 8bcd4d1e28024352b3e4c32c35bd4f86: Deleting consensus metadata
I20260619 20:58:49.566663 17901 catalog_manager.cc:4841] TS 8bcd4d1e28024352b3e4c32c35bd4f86 (127.0.98.2:37131): tablet a350aa5fb9864df8baf64d2775c2da24 (table test-table [id=710490bcd7454ce29741f85b3c889b02]) successfully deleted
W20260619 20:58:49.570564 17900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33113 (error 108)
I20260619 20:58:49.570756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18107
I20260619 20:58:49.571039 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:49.570932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33113 (local address 127.0.0.1:45034)
0619 20:58:49.570973 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:49.571005 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 20:58:49.571139 17900 catalog_manager.cc:4584] TS 8fb947aaca184479ac260d260e77dbf7 (127.0.98.1:33113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a350aa5fb9864df8baf64d2775c2da24: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33113: connect: Connection refused (error 111)
I20260619 20:58:49.575341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18238
I20260619 20:58:49.579561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 17885
2026-06-19T20:58:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (705 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:49 dist-test-slave-8bgz krb5kdc[18408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:49 dist-test-slave-8bgz krb5kdc[18408](info): set up 2 sockets
Jun 19 20:58:49 dist-test-slave-8bgz krb5kdc[18408](info): commencing operation
krb5kdc: starting...
W20260619 20:58:50.618083 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:50 dist-test-slave-8bgz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:58:50.728395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:50.807806 18421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:50.807993 18421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:50.808013 18421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:50.809024 18421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:50.809057 18421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:50.809070 18421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:50.809082 18421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:50.809094 18421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:50.810410 18421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:50.810591 18421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:50.810739 18421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:50.815358 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:50.815363 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:50.815363 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:50.815465 18421 server_base.cc:1034] running on GCE node
I20260619 20:58:50.815814 18421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:50.816035 18421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:50.817171 18421 hybrid_clock.cc:648] HybridClock initialized: now 1781902730817162 us; error 33 us; skew 500 ppm
Jun 19 20:58:50 dist-test-slave-8bgz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:50.819603 18421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:58:50.820039 18421 webserver.cc:460] Webserver started at http://127.0.98.62:38887/ using document root <none> and password file <none>
I20260619 20:58:50.820258 18421 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:50.820323 18421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:50.820472 18421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:50.821499 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "e0530212639b4708acb68dcc05874e06"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "383a2ce8b7301c8f466073afbfc0df0e"
server_key_iv: "bce1f89a6a02d32746d4336f9afdabdb"
server_key_version: "encryptionkey@0"
I20260619 20:58:50.821853 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0530212639b4708acb68dcc05874e06"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "383a2ce8b7301c8f466073afbfc0df0e"
server_key_iv: "bce1f89a6a02d32746d4336f9afdabdb"
server_key_version: "encryptionkey@0"
I20260619 20:58:50.823273 18421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:50.823983 18436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:50.824155 18421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:50.824252 18421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "e0530212639b4708acb68dcc05874e06"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "383a2ce8b7301c8f466073afbfc0df0e"
server_key_iv: "bce1f89a6a02d32746d4336f9afdabdb"
server_key_version: "encryptionkey@0"
I20260619 20:58:50.824338 18421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:50.841276 18421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:50.843267 18421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:50.843463 18421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:50.847584 18421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33761
I20260619 20:58:50.847621 18488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33761 every 8 connection(s)
I20260619 20:58:50.847981 18421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:50.848819 18489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:50.851049 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: Bootstrap starting.
I20260619 20:58:50.851569 18489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:50.851797 18489 log.cc:824] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:50.852448 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: No bootstrap required, opened a new log
I20260619 20:58:50.852571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18421
I20260619 20:58:50.852689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:50.852880 392 external_mini_cluster.cc:1428] Setting key 121006c29d1a36a56c4a598595eaf524
I20260619 20:58:50.854596 18489 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260619 20:58:50.854753 18489 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:50.854796 18489 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0530212639b4708acb68dcc05874e06, State: Initialized, Role: FOLLOWER
I20260619 20:58:50.854926 18489 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260619 20:58:50.855010 18489 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:50.855046 18489 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:50.855101 18489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:50.856105 18489 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260619 20:58:50.856190 18489 leader_election.cc:302] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0530212639b4708acb68dcc05874e06; no voters:
I20260619 20:58:50.856357 18489 leader_election.cc:288] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:50.856403 18494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:50.856529 18494 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [term 1 LEADER]: Becoming Leader. State: Replica: e0530212639b4708acb68dcc05874e06, State: Running, Role: LEADER
I20260619 20:58:50.856622 18489 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:50.856606 18494 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260619 20:58:50.856912 18494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0530212639b4708acb68dcc05874e06. Latest consensus state: current_term: 1 leader_uuid: "e0530212639b4708acb68dcc05874e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } } }
I20260619 20:58:50.856981 18494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:50.857175 18498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:50.857162 18495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0530212639b4708acb68dcc05874e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0530212639b4708acb68dcc05874e06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } } }
I20260619 20:58:50.857363 18495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:50.857637 18498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:50.859740 18498 catalog_manager.cc:1347] Generated new cluster ID: f35141f0edfd44fab87f3d43b605305b
I20260619 20:58:50.859793 18498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:58:50 dist-test-slave-8bgz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:50.864843 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:50.853382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.0.98.62:33761)
0619 20:58:50.853579 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:50.853588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:50.853622 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:58:50.854585 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:50.854591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:50.854645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:50.854797 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:50.855619 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:50.856773 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:50.857835 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:50.858443 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:50.858550 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:50.861225 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:50.861229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:50.861230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:50.861241 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:50.863202 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:50.863582 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:50.863586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:50.863586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:50.863716 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:50.863922 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:50.863923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:50.863923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:50.864220 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:50.864371 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:50.864452 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:58:50.864605 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260619 20:58:50.870038 18498 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:50.870661 18498 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:50.878399 18498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: Generated new TSK 0
I20260619 20:58:50.878535 18498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:58:50.908366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:50.986169 18515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:50.986445 18515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:50.986482 18515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:50.987612 18515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:50.987674 18515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:50.987715 18515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:50.989164 18515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:50.989427 18515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:50.989635 18515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:58:50.994473 18515 server_base.cc:1034] running on GCE node
W20260619 20:58:50.994437 18521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:50.994443 18523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:50.994444 18520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:50.994786 18515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:50.995028 18515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:50.996186 18515 hybrid_clock.cc:648] HybridClock initialized: now 1781902730996161 us; error 35 us; skew 500 ppm
Jun 19 20:58:50 dist-test-slave-8bgz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:50.998821 18515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:58:50.999193 18515 webserver.cc:460] Webserver started at http://127.0.98.1:38085/ using document root <none> and password file <none>
I20260619 20:58:50.999394 18515 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:50.999433 18515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:50.999517 18515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.000495 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "4cdfdca3368e407eb68bce230bb3a006"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "7548fb9659f8b2ed63d1858b70aab85c"
server_key_iv: "b46ca0e4feb931c392b79e5905819bb7"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.000797 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4cdfdca3368e407eb68bce230bb3a006"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "7548fb9659f8b2ed63d1858b70aab85c"
server_key_iv: "b46ca0e4feb931c392b79e5905819bb7"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.001984 18515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:58:51.002707 18530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.002867 18515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.002964 18515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "4cdfdca3368e407eb68bce230bb3a006"
format_stamp: "Formatted at 2026-06-19 20:58:50 on dist-test-slave-8bgz"
server_key: "7548fb9659f8b2ed63d1858b70aab85c"
server_key_iv: "b46ca0e4feb931c392b79e5905819bb7"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.003019 18515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.027861 18515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.029546 18515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.029647 18515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.029848 18515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:51.030151 18515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:51.030184 18515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.030246 18515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:51.030272 18515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.035851 18515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42017
I20260619 20:58:51.035905 18643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42017 every 8 connection(s)
I20260619 20:58:51.036253 18515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:58:51 dist-test-slave-8bgz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:51.043027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18515
I20260619 20:58:51.043151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:51.043378 392 external_mini_cluster.cc:1428] Setting key 5f62d1bc73d298c749fbafa15a809276
I20260619 20:58:51.046172 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.036963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33885 (local address 127.0.98.62:33761)
0619 20:58:51.037033 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.037035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.037249 (+ 214us) server_negotiation.cc:407] Connection header received
0619 20:58:51.038184 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.038186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.038201 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.038342 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.038917 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.039450 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.040410 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.040674 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.040736 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.042959 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.042960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.042961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:51.042966 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.044697 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.045143 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.045144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.045144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.045211 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.045491 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.045492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.045492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.045615 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.045693 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.045958 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.046072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:58:51.046310 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.036915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33761 (local address 127.0.98.1:33885)
0619 20:58:51.037118 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.037156 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.038039 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.038403 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.038407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.038647 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:51.038835 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.038838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.039554 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.039554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.040329 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.040330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.040748 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.040748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.040871 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.041339 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:51.041346 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:51.042862 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:51.044808 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.044815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.044818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.045046 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.045299 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.045300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.045301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.045408 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.045697 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:51.045703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:51.045826 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.045935 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.046080 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260619 20:58:51.046990 18644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33761
I20260619 20:58:51.047101 18644 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:51.047269 18644 heartbeater.cc:511] Master 127.0.98.62:33761 requested a full tablet report, sending...
I20260619 20:58:51.047816 18453 ts_manager.cc:194] Registered new tserver with Master: 4cdfdca3368e407eb68bce230bb3a006 (127.0.98.1:42017)
I20260619 20:58:51.048425 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:58:51.067085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:51.145030 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:51.145203 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:51.145226 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:51.146445 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:51.146484 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:51.146510 18649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:51.147958 18649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:51.148177 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:51.148381 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:51.153123 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.153184 18654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.153162 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.153370 18649 server_base.cc:1034] running on GCE node
I20260619 20:58:51.153581 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:51.153811 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:51.154971 18649 hybrid_clock.cc:648] HybridClock initialized: now 1781902731154954 us; error 28 us; skew 500 ppm
Jun 19 20:58:51 dist-test-slave-8bgz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:51.157577 18649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:58:51.158015 18649 webserver.cc:460] Webserver started at http://127.0.98.2:43311/ using document root <none> and password file <none>
I20260619 20:58:51.158288 18649 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:51.158350 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:51.158492 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.159480 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "e419400e699145fab023e98d75caccfe"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "4c893b4c1bce31936daa719287cf0c4d"
server_key_iv: "b58972b25477a4d82778f3edc3bf5abb"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.159824 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e419400e699145fab023e98d75caccfe"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "4c893b4c1bce31936daa719287cf0c4d"
server_key_iv: "b58972b25477a4d82778f3edc3bf5abb"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.161072 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:51.161733 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.161906 18649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:51.161996 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "e419400e699145fab023e98d75caccfe"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "4c893b4c1bce31936daa719287cf0c4d"
server_key_iv: "b58972b25477a4d82778f3edc3bf5abb"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.162077 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.177727 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.179510 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.179663 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.179872 18649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:51.180146 18649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:51.180197 18649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.180227 18649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:51.180255 18649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.185770 18649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36205
I20260619 20:58:51.185815 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36205 every 8 connection(s)
I20260619 20:58:51.186121 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:51.191162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18649
I20260619 20:58:51.191293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:51.191519 392 external_mini_cluster.cc:1428] Setting key 66a3116631e41bb947805bb8ade52667
Jun 19 20:58:51 dist-test-slave-8bgz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:51.195432 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.186939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57627 (local address 127.0.98.62:33761)
0619 20:58:51.186987 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.186990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.187104 (+ 114us) server_negotiation.cc:407] Connection header received
0619 20:58:51.188013 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.188014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.188026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.188098 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.188624 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.189144 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.190067 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.190317 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.190392 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.192721 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.192723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.192723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:51.192731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.194051 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.194515 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.194516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.194517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.194575 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.194820 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.194821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.194822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.194930 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.195004 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.195264 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.195346 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:58:51.195798 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.186838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33761 (local address 127.0.98.2:57627)
0619 20:58:51.186990 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.187027 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.187900 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.188147 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.188150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.188378 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:51.188547 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.188551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.189226 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.189227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.189989 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.189991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.190400 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.190401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.190496 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.190941 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:51.190950 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:51.192621 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:51.194130 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.194136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.194139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.194424 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.194640 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.194640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.194641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.194751 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.194995 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:51.195000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:51.195131 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.195243 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.195385 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260619 20:58:51.196333 18778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33761
I20260619 20:58:51.196434 18778 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:51.196592 18778 heartbeater.cc:511] Master 127.0.98.62:33761 requested a full tablet report, sending...
I20260619 20:58:51.196993 18453 ts_manager.cc:194] Registered new tserver with Master: e419400e699145fab023e98d75caccfe (127.0.98.2:36205)
I20260619 20:58:51.197377 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57627
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:58:51.216866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:51.297284 18783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:51.297513 18783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:51.297555 18783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:51.298719 18783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:51.298789 18783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:51.298835 18783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:51.300422 18783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:51.300693 18783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:51.300933 18783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:51.305712 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.305754 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.305729 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.305944 18783 server_base.cc:1034] running on GCE node
I20260619 20:58:51.306200 18783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:51.306495 18783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:51.307641 18783 hybrid_clock.cc:648] HybridClock initialized: now 1781902731307633 us; error 28 us; skew 500 ppm
Jun 19 20:58:51 dist-test-slave-8bgz krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:51.309850 18783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:58:51.310292 18783 webserver.cc:460] Webserver started at http://127.0.98.3:39993/ using document root <none> and password file <none>
I20260619 20:58:51.310536 18783 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:51.310578 18783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:51.310729 18783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.311735 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e65997a7331040408185caf1a3cb2735"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "04361d46d10c54206c92ca2d075774e4"
server_key_iv: "3a4a478b4c294fb0b4441a3346ee2b4d"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.312060 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e65997a7331040408185caf1a3cb2735"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "04361d46d10c54206c92ca2d075774e4"
server_key_iv: "3a4a478b4c294fb0b4441a3346ee2b4d"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.313349 18783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:51.313997 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.314145 18783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:51.314255 18783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e65997a7331040408185caf1a3cb2735"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "04361d46d10c54206c92ca2d075774e4"
server_key_iv: "3a4a478b4c294fb0b4441a3346ee2b4d"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.314321 18783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.335246 18783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.337013 18783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.337172 18783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.337389 18783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:51.337693 18783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:51.337728 18783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.337749 18783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:51.337796 18783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.343585 18783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41103
I20260619 20:58:51.343657 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41103 every 8 connection(s)
I20260619 20:58:51.343952 18783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:58:51 dist-test-slave-8bgz krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:51.351593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18783
I20260619 20:58:51.351703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:51.351898 392 external_mini_cluster.cc:1428] Setting key 2e1c376cfb267e0a46b8e0072d7d5ece
I20260619 20:58:51.353117 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.344660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57723 (local address 127.0.98.62:33761)
0619 20:58:51.344734 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.344735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.344900 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:58:51.345799 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.345799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.345821 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.345876 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.346507 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.346994 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.348000 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.348175 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.348224 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.350346 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.350347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.350347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:51.350354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.351607 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.352046 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.352048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.352048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.352108 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.352486 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.352487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.352487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.352625 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.352709 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.352994 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.353056 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 20:58:51.353269 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.344623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33761 (local address 127.0.98.3:57723)
0619 20:58:51.344776 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.344816 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.345715 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.345947 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.345950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.346182 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:51.346437 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.346441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.347094 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.347095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.347920 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.347922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.348260 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.348261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.348373 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.348887 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:51.348895 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:51.350257 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:51.351695 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.351701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.351703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.351952 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.352199 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:51.352200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:51.352200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:51.352332 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:51.352711 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:51.352716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:51.352886 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.352965 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.353075 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260619 20:58:51.353713 18912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33761
I20260619 20:58:51.353785 18912 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:51.353907 18912 heartbeater.cc:511] Master 127.0.98.62:33761 requested a full tablet report, sending...
I20260619 20:58:51.354247 18453 ts_manager.cc:194] Registered new tserver with Master: e65997a7331040408185caf1a3cb2735 (127.0.98.3:41103)
I20260619 20:58:51.354619 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57723
I20260619 20:58:51.363751 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:51.370790 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.365198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.0.98.62:33761)
0619 20:58:51.365252 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.365254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.365333 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:58:51.365386 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.365387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.365395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.365439 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.366001 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.366529 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.367200 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.367375 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.367423 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.368602 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.368603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.368603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:51.368608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.369729 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.370065 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.370066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.370066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.370114 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:51.370332 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:51.370332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:51.370333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:51.370450 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.370531 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.370624 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.370674 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:58:51.372262 18453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:51.373739 18453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:51.380154 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.375896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.0.98.3:41103)
0619 20:58:51.376290 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.376295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.376306 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:58:51.376343 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.376345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.376361 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.376445 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:51.376592 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377663 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379188 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.379740 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379828 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.379885 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.380031 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.380069 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":87,"threads_started":1}
I20260619 20:58:51.380164 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.375991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42017 (local address 127.0.0.1:33992)
0619 20:58:51.376582 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.376593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.376646 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.376827 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.376828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.376830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:51.376862 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.376863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377998 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.377999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.378897 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.378900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.379901 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.379963 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.379967 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.380015 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.380060 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":3,"thread_start_us":39,"threads_started":1}
I20260619 20:58:51.380172 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.375787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41103 (local address 127.0.0.1:59378)
0619 20:58:51.375949 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.375986 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.376053 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.376464 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.376467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.376469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:51.376535 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.376537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377763 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.377764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.379101 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.379104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.379829 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.379920 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.379933 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.380004 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.380064 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":73,"thread_start_us":80,"threads_started":1}
I20260619 20:58:51.380544 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.376132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.1:42017)
0619 20:58:51.376303 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.376307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.376641 (+ 334us) server_negotiation.cc:407] Connection header received
0619 20:58:51.376722 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.376724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.376739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.376812 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:51.376917 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377902 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.378986 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.379741 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379811 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.379866 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.380412 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.380453 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260619 20:58:51.380766 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.375793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36205 (local address 127.0.0.1:56654)
0619 20:58:51.376109 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.376119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.376159 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.376486 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.376487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.376489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:51.376548 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.376550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377711 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.377713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.379284 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.379286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.380192 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.380193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.380460 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.380463 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.380511 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.380548 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":54,"thread_start_us":57,"threads_started":1}
I20260619 20:58:51.381062 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.376117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.2:36205)
0619 20:58:51.376319 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.376323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.376341 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:58:51.376376 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.376378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.376389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.376485 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:51.376636 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.377604 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.379357 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.380052 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.380110 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.380159 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.380958 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.380998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260619 20:58:51.380962 18846 tablet_service.cc:1467] Processing CreateTablet for tablet 3eac7cf6c16c491c947614b13ade98de (DEFAULT_TABLE table=test-table [id=12f7570a9e774553a2ee0ea28b610067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:51.381117 18578 tablet_service.cc:1467] Processing CreateTablet for tablet 3eac7cf6c16c491c947614b13ade98de (DEFAULT_TABLE table=test-table [id=12f7570a9e774553a2ee0ea28b610067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:51.381300 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eac7cf6c16c491c947614b13ade98de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:51.381392 18578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eac7cf6c16c491c947614b13ade98de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:51.384560 18927 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: Bootstrap starting.
I20260619 20:58:51.384922 18712 tablet_service.cc:1467] Processing CreateTablet for tablet 3eac7cf6c16c491c947614b13ade98de (DEFAULT_TABLE table=test-table [id=12f7570a9e774553a2ee0ea28b610067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:51.385176 18927 tablet_bootstrap.cc:654] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:51.385224 18712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eac7cf6c16c491c947614b13ade98de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:51.385643 18927 log.cc:824] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:51.385970 18926 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Bootstrap starting.
I20260619 20:58:51.386401 18927 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: No bootstrap required, opened a new log
I20260619 20:58:51.386471 18927 ts_tablet_manager.cc:1397] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:58:51.386651 18926 tablet_bootstrap.cc:654] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:51.387194 18926 log.cc:824] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:51.387952 18926 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: No bootstrap required, opened a new log
I20260619 20:58:51.388023 18926 ts_tablet_manager.cc:1397] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:58:51.388151 18930 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Bootstrap starting.
I20260619 20:58:51.388530 18927 raft_consensus.cc:348] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.388715 18927 raft_consensus.cc:374] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:51.388751 18927 raft_consensus.cc:729] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdfdca3368e407eb68bce230bb3a006, State: Initialized, Role: FOLLOWER
I20260619 20:58:51.388810 18930 tablet_bootstrap.cc:654] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:51.388870 18927 consensus_queue.cc:260] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.389130 18927 ts_tablet_manager.cc:1428] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:51.389257 18644 heartbeater.cc:503] Master 127.0.98.62:33761 was elected leader, sending a full tablet report...
I20260619 20:58:51.389361 18930 log.cc:824] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:51.389950 18926 raft_consensus.cc:348] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.390134 18926 raft_consensus.cc:374] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:51.390167 18930 tablet_bootstrap.cc:492] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: No bootstrap required, opened a new log
I20260619 20:58:51.390190 18926 raft_consensus.cc:729] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65997a7331040408185caf1a3cb2735, State: Initialized, Role: FOLLOWER
I20260619 20:58:51.390249 18930 ts_tablet_manager.cc:1397] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:51.390319 18926 consensus_queue.cc:260] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.390565 18926 ts_tablet_manager.cc:1428] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:58:51.390630 18912 heartbeater.cc:503] Master 127.0.98.62:33761 was elected leader, sending a full tablet report...
I20260619 20:58:51.391985 18930 raft_consensus.cc:348] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.392134 18930 raft_consensus.cc:374] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:51.392182 18930 raft_consensus.cc:729] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e419400e699145fab023e98d75caccfe, State: Initialized, Role: FOLLOWER
I20260619 20:58:51.392284 18930 consensus_queue.cc:260] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.392539 18930 ts_tablet_manager.cc:1428] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:58:51.392565 18778 heartbeater.cc:503] Master 127.0.98.62:33761 was elected leader, sending a full tablet report...
W20260619 20:58:51.436767 18779 tablet.cc:2369] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:51.468868 18933 raft_consensus.cc:482] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:51.468990 18933 raft_consensus.cc:504] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.469322 18933 leader_election.cc:288] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e419400e699145fab023e98d75caccfe (127.0.98.2:36205), 4cdfdca3368e407eb68bce230bb3a006 (127.0.98.1:42017)
I20260619 20:58:51.473353 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.469599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42321 (local address 127.0.98.2:36205)
0619 20:58:51.469675 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.469677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.469687 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:51.469717 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.469723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.469734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.469785 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:51.469994 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.470811 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.472369 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.472972 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.473050 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.473061 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.473245 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.473283 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 20:58:51.473353 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.469599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40691 (local address 127.0.98.1:42017)
0619 20:58:51.469646 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.469648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.469836 (+ 188us) server_negotiation.cc:407] Connection header received
0619 20:58:51.469873 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.469875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.469885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.469927 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:51.470088 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.470809 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.472369 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.472957 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.473056 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.473066 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.473245 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.473282 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:58:51.473357 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.469515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36205 (local address 127.0.98.3:42321)
0619 20:58:51.469570 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.469583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.469624 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.469814 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.469815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.469818 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:51.469893 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.469899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.470896 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.470896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.472267 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.472269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.473066 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.473067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.473162 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.473164 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.473223 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.473263 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":119}
I20260619 20:58:51.473367 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.469553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42017 (local address 127.0.98.3:40691)
0619 20:58:51.469734 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.469742 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.469811 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.469951 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.469952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.469953 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:51.470019 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.470020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.470896 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.470896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.472267 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.472269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.473073 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.473074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.473160 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.473164 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.473218 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.473263 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":57,"thread_start_us":114,"threads_started":1}
I20260619 20:58:51.473805 18598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eac7cf6c16c491c947614b13ade98de" candidate_uuid: "e65997a7331040408185caf1a3cb2735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdfdca3368e407eb68bce230bb3a006" is_pre_election: true
I20260619 20:58:51.474033 18598 raft_consensus.cc:2454] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65997a7331040408185caf1a3cb2735 in term 0.
I20260619 20:58:51.473999 18732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eac7cf6c16c491c947614b13ade98de" candidate_uuid: "e65997a7331040408185caf1a3cb2735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e419400e699145fab023e98d75caccfe" is_pre_election: true
I20260619 20:58:51.474149 18732 raft_consensus.cc:2454] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65997a7331040408185caf1a3cb2735 in term 0.
I20260619 20:58:51.474344 18802 leader_election.cc:302] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cdfdca3368e407eb68bce230bb3a006, e65997a7331040408185caf1a3cb2735; no voters:
I20260619 20:58:51.474485 18933 raft_consensus.cc:2781] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:51.474576 18933 raft_consensus.cc:482] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:51.474607 18933 raft_consensus.cc:3037] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:51.475417 18933 raft_consensus.cc:504] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.475566 18933 leader_election.cc:288] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [CANDIDATE]: Term 1 election: Requested vote from peers e419400e699145fab023e98d75caccfe (127.0.98.2:36205), 4cdfdca3368e407eb68bce230bb3a006 (127.0.98.1:42017)
I20260619 20:58:51.475718 18598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eac7cf6c16c491c947614b13ade98de" candidate_uuid: "e65997a7331040408185caf1a3cb2735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdfdca3368e407eb68bce230bb3a006"
I20260619 20:58:51.475782 18732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eac7cf6c16c491c947614b13ade98de" candidate_uuid: "e65997a7331040408185caf1a3cb2735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e419400e699145fab023e98d75caccfe"
I20260619 20:58:51.475802 18598 raft_consensus.cc:3037] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:51.475865 18732 raft_consensus.cc:3037] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:51.476457 18598 raft_consensus.cc:2454] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65997a7331040408185caf1a3cb2735 in term 1.
I20260619 20:58:51.476534 18732 raft_consensus.cc:2454] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65997a7331040408185caf1a3cb2735 in term 1.
I20260619 20:58:51.476662 18802 leader_election.cc:302] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cdfdca3368e407eb68bce230bb3a006, e65997a7331040408185caf1a3cb2735; no voters:
I20260619 20:58:51.476783 18933 raft_consensus.cc:2781] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:51.476956 18933 raft_consensus.cc:686] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 1 LEADER]: Becoming Leader. State: Replica: e65997a7331040408185caf1a3cb2735, State: Running, Role: LEADER
I20260619 20:58:51.477087 18933 consensus_queue.cc:237] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } }
I20260619 20:58:51.477885 18452 catalog_manager.cc:5495] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 reported cstate change: term changed from 0 to 1, leader changed from <none> to e65997a7331040408185caf1a3cb2735 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e65997a7331040408185caf1a3cb2735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e65997a7331040408185caf1a3cb2735" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 } health_report { overall_health: UNKNOWN } } }
I20260619 20:58:51.502468 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.500343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.98.3:41103)
0619 20:58:51.500390 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.500393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.500401 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:58:51.500440 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.500441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.500456 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.500556 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:51.500609 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.501063 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.501742 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.501915 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.501963 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.502085 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:51.502164 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:51.502251 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.502338 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.502346 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:58:51.504160 18598 raft_consensus.cc:1261] T 3eac7cf6c16c491c947614b13ade98de P 4cdfdca3368e407eb68bce230bb3a006 [term 1 FOLLOWER]: Refusing update from remote peer e65997a7331040408185caf1a3cb2735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:51.504469 18732 raft_consensus.cc:1261] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 1 FOLLOWER]: Refusing update from remote peer e65997a7331040408185caf1a3cb2735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:51.504479 18936 consensus_queue.cc:1035] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdfdca3368e407eb68bce230bb3a006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:51.504743 18936 consensus_queue.cc:1035] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e419400e699145fab023e98d75caccfe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:51.509192 18453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:51.509286 18453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:51.509999 18453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0530212639b4708acb68dcc05874e06: Sending DeleteTablet for 3 replicas of tablet 3eac7cf6c16c491c947614b13ade98de
I20260619 20:58:51.510349 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eac7cf6c16c491c947614b13ade98de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59378
I20260619 20:58:51.510368 18712 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eac7cf6c16c491c947614b13ade98de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56654
I20260619 20:58:51.510556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18515
I20260619 20:58:51.510658 18578 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eac7cf6c16c491c947614b13ade98de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33992
I20260619 20:58:51.510598 18948 tablet_replica.cc:330] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: stopping tablet replica
I20260619 20:58:51.511096 18949 tablet_replica.cc:330] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: stopping tablet replica
I20260619 20:58:51.511183 18949 raft_consensus.cc:2229] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:51.511183 18948 raft_consensus.cc:2229] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:51.511260 18948 raft_consensus.cc:2258] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:51.511271 18949 raft_consensus.cc:2258] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:51.511688 18949 ts_tablet_manager.cc:1905] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:51.511688 18948 ts_tablet_manager.cc:1905] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:51.513336 18949 ts_tablet_manager.cc:1918] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:51.513336 18948 ts_tablet_manager.cc:1918] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:51.513411 18949 log.cc:1196] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/3eac7cf6c16c491c947614b13ade98de
I20260619 20:58:51.513411 18948 log.cc:1196] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/3eac7cf6c16c491c947614b13ade98de
I20260619 20:58:51.513701 18949 ts_tablet_manager.cc:1939] T 3eac7cf6c16c491c947614b13ade98de P e419400e699145fab023e98d75caccfe: Deleting consensus metadata
I20260619 20:58:51.513721 18948 ts_tablet_manager.cc:1939] T 3eac7cf6c16c491c947614b13ade98de P e65997a7331040408185caf1a3cb2735: Deleting consensus metadata
I20260619 20:58:51.514124 18438 catalog_manager.cc:4841] TS e419400e699145fab023e98d75caccfe (127.0.98.2:36205): tablet 3eac7cf6c16c491c947614b13ade98de (table test-table [id=12f7570a9e774553a2ee0ea28b610067]) successfully deleted
I20260619 20:58:51.514191 18440 catalog_manager.cc:4841] TS e65997a7331040408185caf1a3cb2735 (127.0.98.3:41103): tablet 3eac7cf6c16c491c947614b13ade98de (table test-table [id=12f7570a9e774553a2ee0ea28b610067]) successfully deleted
W20260619 20:58:51.516027 18440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42017 (error 108)
I20260619 20:58:51.516566 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.516431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42017 (local address 127.0.0.1:34004)
0619 20:58:51.516491 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.516530 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 20:58:51.516757 18440 catalog_manager.cc:4584] TS 4cdfdca3368e407eb68bce230bb3a006 (127.0.98.1:42017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3eac7cf6c16c491c947614b13ade98de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42017: connect: Connection refused (error 111)
I20260619 20:58:51.516858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18649
I20260619 20:58:51.521564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18783
I20260619 20:58:51.526716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18421
2026-06-19T20:58:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T20:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:51Z Disabled control of system clock
I20260619 20:58:51.552790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:51.630805 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:51.631004 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:51.631024 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:51.632066 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:51.632097 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:51.632109 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:51.632121 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:51.632133 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:51.633342 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:51.633508 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:51.633713 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:51.638324 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.638458 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.638499 18954 server_base.cc:1034] running on GCE node
W20260619 20:58:51.638299 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.638793 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:51.639081 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:51.640249 18954 hybrid_clock.cc:648] HybridClock initialized: now 1781902731640233 us; error 26 us; skew 500 ppm
I20260619 20:58:51.641357 18954 webserver.cc:460] Webserver started at http://127.0.98.62:45015/ using document root <none> and password file <none>
I20260619 20:58:51.641590 18954 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:51.641659 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:51.641805 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.642890 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "95b5f618168b4f34942efe028c7dfd06"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "e27723ef24068e5c672cdb602798e942"
server_key_iv: "9cf2ab4aff16732f4dbe346e49cf44c8"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.643246 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "95b5f618168b4f34942efe028c7dfd06"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "e27723ef24068e5c672cdb602798e942"
server_key_iv: "9cf2ab4aff16732f4dbe346e49cf44c8"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.644481 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:58:51.645179 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.645354 18954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:51.645453 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "95b5f618168b4f34942efe028c7dfd06"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "e27723ef24068e5c672cdb602798e942"
server_key_iv: "9cf2ab4aff16732f4dbe346e49cf44c8"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.645535 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.655501 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.655762 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.655902 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.659574 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36539
I20260619 20:58:51.659603 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36539 every 8 connection(s)
I20260619 20:58:51.659909 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:51.660812 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:51.663000 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: Bootstrap starting.
I20260619 20:58:51.663509 19021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:51.663733 19021 log.cc:824] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:51.664317 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: No bootstrap required, opened a new log
I20260619 20:58:51.665788 19021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 20:58:51.665923 19021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:51.665963 19021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b5f618168b4f34942efe028c7dfd06, State: Initialized, Role: FOLLOWER
I20260619 20:58:51.666102 19021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 20:58:51.666261 19021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:51.666311 19021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:51.666343 19021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:51.666651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 18954
I20260619 20:58:51.666764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:51.666832 19021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 20:58:51.666946 19021 leader_election.cc:302] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b5f618168b4f34942efe028c7dfd06; no voters:
I20260619 20:58:51.666975 392 external_mini_cluster.cc:1428] Setting key c85d09c50e2ca4764d06f14a0db2c368
I20260619 20:58:51.667124 19021 leader_election.cc:288] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:51.667158 19024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:51.667275 19024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [term 1 LEADER]: Becoming Leader. State: Replica: 95b5f618168b4f34942efe028c7dfd06, State: Running, Role: LEADER
I20260619 20:58:51.667374 19021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:51.667349 19024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 20:58:51.667747 19024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b5f618168b4f34942efe028c7dfd06. Latest consensus state: current_term: 1 leader_uuid: "95b5f618168b4f34942efe028c7dfd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 20:58:51.667851 19024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:51.668074 19025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b5f618168b4f34942efe028c7dfd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5f618168b4f34942efe028c7dfd06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 20:58:51.668167 19025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:51.668912 19033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:51.669588 19033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:51.671098 19033 catalog_manager.cc:1347] Generated new cluster ID: abf7976dd80143f58f833b021bda9eb7
I20260619 20:58:51.671152 19033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:51.672991 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.667547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.0.98.62:36539)
0619 20:58:51.667759 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.667782 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.667820 (+ 38us) server_negotiation.cc:407] Connection header received
0619 20:58:51.668156 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.668164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.668234 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.668337 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.668721 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.670072 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.671101 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.671559 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.671685 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.672387 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.672390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.672390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:51.672561 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.672591 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:51.672601 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.672682 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.672755 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.672828 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260619 20:58:51.686653 19033 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:51.687162 19033 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:51.691736 19033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: Generated new TSK 0
I20260619 20:58:51.691897 19033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:51.695485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:51.780795 19045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:51.781000 19045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:51.781020 19045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:51.782150 19045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:51.782196 19045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:51.782263 19045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:51.783545 19045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:51.783769 19045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:51.784039 19045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:51.788805 19051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.788811 19053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.788873 19050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.789130 19045 server_base.cc:1034] running on GCE node
I20260619 20:58:51.789404 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:51.789633 19045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:51.790794 19045 hybrid_clock.cc:648] HybridClock initialized: now 1781902731790779 us; error 31 us; skew 500 ppm
I20260619 20:58:51.792093 19045 webserver.cc:460] Webserver started at http://127.0.98.1:46247/ using document root <none> and password file <none>
I20260619 20:58:51.792335 19045 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:51.792402 19045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:51.792553 19045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.793558 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb03c1d5fde344aaad142f77f76dd585"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8d41330db797a8d66bcbe5c714ab6f16"
server_key_iv: "4de205158a3afd603804e459e3c806a9"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.793906 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb03c1d5fde344aaad142f77f76dd585"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8d41330db797a8d66bcbe5c714ab6f16"
server_key_iv: "4de205158a3afd603804e459e3c806a9"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.795236 19045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:51.795979 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.796150 19045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:51.796255 19045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "fb03c1d5fde344aaad142f77f76dd585"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8d41330db797a8d66bcbe5c714ab6f16"
server_key_iv: "4de205158a3afd603804e459e3c806a9"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.796321 19045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.813400 19045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.813687 19045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.813859 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.814061 19045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:51.814419 19045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:51.814451 19045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.814502 19045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:51.814525 19045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.820212 19045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41577
I20260619 20:58:51.820259 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41577 every 8 connection(s)
I20260619 20:58:51.820565 19045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:51.825207 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.821245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33281 (local address 127.0.98.62:36539)
0619 20:58:51.821317 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.821319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.821519 (+ 200us) server_negotiation.cc:407] Connection header received
0619 20:58:51.821964 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.821965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.821975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.822044 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.822706 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.823189 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.824084 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.824261 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.824310 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.825009 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.825010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.825010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:51.825022 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.825037 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:51.825041 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.825083 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.825127 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.825150 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:58:51.825436 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.821236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:33281)
0619 20:58:51.821396 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.821435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.821887 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.822096 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.822099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.822106 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:51.822632 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.822637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.823273 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.823273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.824010 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.824011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.824330 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.824330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.824431 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.824885 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:51.824886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:51.824911 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:51.824931 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:51.825081 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:51.825083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:51.825085 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.825128 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.825205 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260619 20:58:51.825968 19173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 20:58:51.826068 19173 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:51.826265 19173 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 20:58:51.826767 18985 ts_manager.cc:194] Registered new tserver with Master: fb03c1d5fde344aaad142f77f76dd585 (127.0.98.1:41577)
I20260619 20:58:51.827390 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33281
I20260619 20:58:51.830585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19045
I20260619 20:58:51.830668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:51.830802 392 external_mini_cluster.cc:1428] Setting key a76b19279dbd82fc41e1cfed3e81453c
I20260619 20:58:51.831657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:51.911588 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:51.911787 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:51.911809 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:51.912951 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:51.912992 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:51.913017 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:51.914410 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:51.914641 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:51.914942 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:51.919862 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:51.919843 19181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.919909 19176 server_base.cc:1034] running on GCE node
W20260619 20:58:51.919872 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:51.920250 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:51.920500 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:51.921666 19176 hybrid_clock.cc:648] HybridClock initialized: now 1781902731921660 us; error 36 us; skew 500 ppm
I20260619 20:58:51.922874 19176 webserver.cc:460] Webserver started at http://127.0.98.2:34717/ using document root <none> and password file <none>
I20260619 20:58:51.923121 19176 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:51.923197 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:51.923333 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:51.924333 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef2edaf9eaab479aa94ba298fb06aeec"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8202b1b925bd417ce1b27ea0740b0513"
server_key_iv: "7e5019f2cb573849a6d568505e3748d0"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.924683 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef2edaf9eaab479aa94ba298fb06aeec"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8202b1b925bd417ce1b27ea0740b0513"
server_key_iv: "7e5019f2cb573849a6d568505e3748d0"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.926010 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:51.926767 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.926928 19176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.927031 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "ef2edaf9eaab479aa94ba298fb06aeec"
format_stamp: "Formatted at 2026-06-19 20:58:51 on dist-test-slave-8bgz"
server_key: "8202b1b925bd417ce1b27ea0740b0513"
server_key_iv: "7e5019f2cb573849a6d568505e3748d0"
server_key_version: "encryptionkey@0"
I20260619 20:58:51.927117 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:51.943795 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:51.944080 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:51.944231 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:51.944442 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:51.944741 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:51.944796 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.944836 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:51.944868 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:51.950630 19176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38311
I20260619 20:58:51.950707 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38311 every 8 connection(s)
I20260619 20:58:51.951021 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:51.956005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19176
I20260619 20:58:51.956102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:51.956286 392 external_mini_cluster.cc:1428] Setting key a8289b930f976b56cb98548a5e212f39
I20260619 20:58:51.956633 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.951741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33375 (local address 127.0.98.62:36539)
0619 20:58:51.951796 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:51.951798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:51.952020 (+ 222us) server_negotiation.cc:407] Connection header received
0619 20:58:51.952564 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:51.952565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:51.952579 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:51.952637 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:51.953346 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.954055 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.955281 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.955469 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.955531 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.956355 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:51.956357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:51.956357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:51.956363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:51.956381 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:51.956386 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:51.956448 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:51.956524 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:58:51.956552 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:58:51.956889 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:51.951709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:33375)
0619 20:58:51.951890 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:51.951922 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:51.952428 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:51.952714 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:51.952717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:51.952727 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:51.953219 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.953223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.954159 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.954160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.955201 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:51.955204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:51.955552 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:51.955553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:51.955658 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:51.956216 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:51.956216 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:51.956243 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:51.956265 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:51.956453 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:51.956457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:51.956460 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:51.956542 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:58:51.956646 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260619 20:58:51.957278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:58:51.958736 19304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 20:58:51.958842 19304 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:51.959015 19304 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 20:58:51.959407 18985 ts_manager.cc:194] Registered new tserver with Master: ef2edaf9eaab479aa94ba298fb06aeec (127.0.98.2:38311)
I20260619 20:58:51.959918 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33375
W20260619 20:58:52.037712 19307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:52.037901 19307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:52.037922 19307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:52.039078 19307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:52.039119 19307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:52.039142 19307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:52.040409 19307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:52.040588 19307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:52.040822 19307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:52.045573 19315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:52.045746 19307 server_base.cc:1034] running on GCE node
W20260619 20:58:52.045604 19313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:52.045799 19312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:52.046036 19307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:52.046345 19307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:52.047547 19307 hybrid_clock.cc:648] HybridClock initialized: now 1781902732047528 us; error 43 us; skew 500 ppm
I20260619 20:58:52.051932 19307 webserver.cc:460] Webserver started at http://127.0.98.3:35131/ using document root <none> and password file <none>
I20260619 20:58:52.052311 19307 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:52.052363 19307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:52.052546 19307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:52.053649 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4"
format_stamp: "Formatted at 2026-06-19 20:58:52 on dist-test-slave-8bgz"
server_key: "c7cdc0be722090e83f812df18cf1693b"
server_key_iv: "41f35784ed901bcfc89064e7c2f7554e"
server_key_version: "encryptionkey@0"
I20260619 20:58:52.054018 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4"
format_stamp: "Formatted at 2026-06-19 20:58:52 on dist-test-slave-8bgz"
server_key: "c7cdc0be722090e83f812df18cf1693b"
server_key_iv: "41f35784ed901bcfc89064e7c2f7554e"
server_key_version: "encryptionkey@0"
I20260619 20:58:52.055346 19307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:52.056098 19321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:52.056301 19307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:52.056387 19307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4"
format_stamp: "Formatted at 2026-06-19 20:58:52 on dist-test-slave-8bgz"
server_key: "c7cdc0be722090e83f812df18cf1693b"
server_key_iv: "41f35784ed901bcfc89064e7c2f7554e"
server_key_version: "encryptionkey@0"
I20260619 20:58:52.056466 19307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:52.075860 19307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:52.076179 19307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:52.076333 19307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:52.076556 19307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:52.076910 19307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:52.076948 19307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:52.076982 19307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:52.077033 19307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:52.083222 19307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33491
I20260619 20:58:52.083290 19434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33491 every 8 connection(s)
I20260619 20:58:52.083602 19307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:52.088688 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.084374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57041 (local address 127.0.98.62:36539)
0619 20:58:52.084439 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.084441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.084648 (+ 207us) server_negotiation.cc:407] Connection header received
0619 20:58:52.085129 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.085130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.085139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.085204 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:52.085856 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.086426 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.087389 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.087565 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.087614 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.088367 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:52.088368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:52.088369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:52.088373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:52.088387 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:52.088392 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:52.088454 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.088582 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.088617 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:58:52.088797 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.084337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:57041)
0619 20:58:52.084519 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.084557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.085029 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.085256 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.085259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.085266 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:52.085753 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.085757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.086508 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.086509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.087312 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.087314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.087639 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.087639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.087747 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.088206 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:52.088207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:52.088234 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:52.088257 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:52.088474 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:52.088478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:52.088481 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.088550 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.088632 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260619 20:58:52.089452 19435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 20:58:52.089557 19435 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:52.089757 19435 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 20:58:52.090143 18985 ts_manager.cc:194] Registered new tserver with Master: e3fdde4821bd4e969ea5ed5f8a5f8dd4 (127.0.98.3:33491)
I20260619 20:58:52.090528 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57041
I20260619 20:58:52.092343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19307
I20260619 20:58:52.092419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:52.092552 392 external_mini_cluster.cc:1428] Setting key ede7ea94580abac215ab07dba6db4311
I20260619 20:58:52.093420 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:52.097280 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.094706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.62:36539)
0619 20:58:52.094744 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.094746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.094817 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:58:52.094874 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.094875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.094884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.094930 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:52.095066 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.095492 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.096218 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.096387 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.096433 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.097069 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:52.097070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:52.097070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:52.097073 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:52.097083 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:52.097086 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:52.097150 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.097224 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.097233 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:58:52.098883 18985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:52.100371 18985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:52.106417 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41577 (local address 127.0.0.1:41042)
0619 20:58:52.102738 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.102747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.102806 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.102983 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.102984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.102986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:52.103027 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.103029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.104080 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.104081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.105238 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.105241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.106076 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.106077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.106156 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106169 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.106287 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.106344 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260619 20:58:52.106426 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.1:41577)
0619 20:58:52.102754 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.102759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.102791 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:58:52.102878 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.102880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.102893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.102986 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:52.103303 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.103996 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.105332 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.105988 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.106074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106158 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.106307 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.106346 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260619 20:58:52.106451 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.0.98.3:33491)
0619 20:58:52.102658 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.102671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.102682 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:58:52.102817 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.102819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.102831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.102908 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:52.103086 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.104135 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.105195 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.105898 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.105945 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106026 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.106332 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.106375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260619 20:58:52.106690 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33491 (local address 127.0.0.1:48418)
0619 20:58:52.102585 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.102625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.102697 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.102925 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.102928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.102930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:52.102995 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.102997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.104228 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.104229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.105071 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.105074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.106087 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.106087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.106166 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106170 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.106570 (+ 400us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.106614 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":56,"threads_started":1}
I20260619 20:58:52.106899 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38311 (local address 127.0.0.1:45068)
0619 20:58:52.102992 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.103001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.103062 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.103244 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.103245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.103247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:52.103279 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.103280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.104480 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.104481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.105583 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.105590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.106685 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.106686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.106748 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106751 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.106806 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.106842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260619 20:58:52.106935 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.102493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.98.2:38311)
0619 20:58:52.102728 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.102732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.103063 (+ 331us) server_negotiation.cc:407] Connection header received
0619 20:58:52.103131 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.103133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.103148 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.103230 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:52.103365 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.104394 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.105678 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.106502 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.106572 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.106652 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.106821 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.106856 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260619 20:58:52.107476 19369 tablet_service.cc:1467] Processing CreateTablet for tablet 11af2518d022499489ea8da291505713 (DEFAULT_TABLE table=test-table [id=0c220e573488439397252d745c690da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:52.107492 19107 tablet_service.cc:1467] Processing CreateTablet for tablet 11af2518d022499489ea8da291505713 (DEFAULT_TABLE table=test-table [id=0c220e573488439397252d745c690da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:52.107693 19238 tablet_service.cc:1467] Processing CreateTablet for tablet 11af2518d022499489ea8da291505713 (DEFAULT_TABLE table=test-table [id=0c220e573488439397252d745c690da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:52.107820 19107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11af2518d022499489ea8da291505713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:52.107820 19369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11af2518d022499489ea8da291505713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:52.108052 19238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11af2518d022499489ea8da291505713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:52.111069 19451 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Bootstrap starting.
I20260619 20:58:52.111172 19449 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Bootstrap starting.
I20260619 20:58:52.111644 19451 tablet_bootstrap.cc:654] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:52.111706 19449 tablet_bootstrap.cc:654] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:52.111958 19451 log.cc:824] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:52.111995 19449 log.cc:824] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:52.112293 19450 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Bootstrap starting.
I20260619 20:58:52.112694 19451 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: No bootstrap required, opened a new log
I20260619 20:58:52.112706 19449 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: No bootstrap required, opened a new log
I20260619 20:58:52.112771 19451 ts_tablet_manager.cc:1397] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:52.112812 19450 tablet_bootstrap.cc:654] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:52.112834 19449 ts_tablet_manager.cc:1397] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:52.113083 19450 log.cc:824] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:52.113729 19450 tablet_bootstrap.cc:492] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: No bootstrap required, opened a new log
I20260619 20:58:52.113792 19450 ts_tablet_manager.cc:1397] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:52.115350 19451 raft_consensus.cc:348] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.115350 19449 raft_consensus.cc:348] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.115525 19449 raft_consensus.cc:374] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:52.115520 19451 raft_consensus.cc:374] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:52.115558 19451 raft_consensus.cc:729] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fdde4821bd4e969ea5ed5f8a5f8dd4, State: Initialized, Role: FOLLOWER
I20260619 20:58:52.115558 19449 raft_consensus.cc:729] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2edaf9eaab479aa94ba298fb06aeec, State: Initialized, Role: FOLLOWER
I20260619 20:58:52.115677 19449 consensus_queue.cc:260] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.115711 19451 consensus_queue.cc:260] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.115772 19450 raft_consensus.cc:348] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.115933 19451 ts_tablet_manager.cc:1428] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:58:52.115933 19449 ts_tablet_manager.cc:1428] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:52.115933 19450 raft_consensus.cc:374] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:52.115981 19450 raft_consensus.cc:729] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb03c1d5fde344aaad142f77f76dd585, State: Initialized, Role: FOLLOWER
I20260619 20:58:52.116051 19304 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 20:58:52.116124 19435 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 20:58:52.116124 19450 consensus_queue.cc:260] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.116386 19450 ts_tablet_manager.cc:1428] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:58:52.116596 19173 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
W20260619 20:58:52.201558 19305 tablet.cc:2369] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:58:52.321197 19174 tablet.cc:2369] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:58:52.334184 19436 tablet.cc:2369] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:52.391016 19457 raft_consensus.cc:482] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:52.391129 19457 raft_consensus.cc:504] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.391492 19457 leader_election.cc:288] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fdde4821bd4e969ea5ed5f8a5f8dd4 (127.0.98.3:33491), ef2edaf9eaab479aa94ba298fb06aeec (127.0.98.2:38311)
I20260619 20:58:52.394621 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.391754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57983 (local address 127.0.98.2:38311)
0619 20:58:52.391802 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.391804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.391883 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:58:52.391965 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.391966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.391976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.392030 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:52.392217 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.392775 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.393735 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.394288 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.394357 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.394366 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.394502 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.394546 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:58:52.394641 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.391674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38311 (local address 127.0.98.1:57983)
0619 20:58:52.391800 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.391811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.391895 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.392057 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.392058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.392061 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:52.392143 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.392145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.392853 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.392854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.393653 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.393656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.394370 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.394371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.394444 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.394448 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.394496 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.394530 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260619 20:58:52.394703 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.391851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49679 (local address 127.0.98.3:33491)
0619 20:58:52.391903 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.391905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.392048 (+ 143us) server_negotiation.cc:407] Connection header received
0619 20:58:52.392087 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.392087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.392097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.392186 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:52.392378 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.392844 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.393825 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.394340 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.394411 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.394420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.394592 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.394630 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:58:52.395232 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.391674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33491 (local address 127.0.98.1:49679)
0619 20:58:52.391969 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.391979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:52.392041 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:52.392215 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:52.392216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:52.392217 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:52.392284 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.392286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.392916 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.392917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.393739 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:52.393741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.394426 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.394427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:52.394517 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.394522 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:58:52.394576 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:58:52.394610 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":12,"thread_start_us":59,"threads_started":1}
I20260619 20:58:52.395352 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11af2518d022499489ea8da291505713" candidate_uuid: "fb03c1d5fde344aaad142f77f76dd585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" is_pre_election: true
I20260619 20:58:52.395515 19258 raft_consensus.cc:2454] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb03c1d5fde344aaad142f77f76dd585 in term 0.
I20260619 20:58:52.395602 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11af2518d022499489ea8da291505713" candidate_uuid: "fb03c1d5fde344aaad142f77f76dd585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" is_pre_election: true
I20260619 20:58:52.395743 19063 leader_election.cc:302] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef2edaf9eaab479aa94ba298fb06aeec, fb03c1d5fde344aaad142f77f76dd585; no voters:
I20260619 20:58:52.395762 19389 raft_consensus.cc:2454] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb03c1d5fde344aaad142f77f76dd585 in term 0.
I20260619 20:58:52.395871 19457 raft_consensus.cc:2781] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:52.395923 19457 raft_consensus.cc:482] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:52.395939 19457 raft_consensus.cc:3037] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:52.396586 19457 raft_consensus.cc:504] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.396718 19457 leader_election.cc:288] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [CANDIDATE]: Term 1 election: Requested vote from peers e3fdde4821bd4e969ea5ed5f8a5f8dd4 (127.0.98.3:33491), ef2edaf9eaab479aa94ba298fb06aeec (127.0.98.2:38311)
I20260619 20:58:52.396819 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11af2518d022499489ea8da291505713" candidate_uuid: "fb03c1d5fde344aaad142f77f76dd585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4"
I20260619 20:58:52.396862 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11af2518d022499489ea8da291505713" candidate_uuid: "fb03c1d5fde344aaad142f77f76dd585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2edaf9eaab479aa94ba298fb06aeec"
I20260619 20:58:52.396889 19389 raft_consensus.cc:3037] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:52.396925 19258 raft_consensus.cc:3037] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:52.397785 19389 raft_consensus.cc:2454] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb03c1d5fde344aaad142f77f76dd585 in term 1.
I20260619 20:58:52.397785 19258 raft_consensus.cc:2454] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb03c1d5fde344aaad142f77f76dd585 in term 1.
I20260619 20:58:52.397951 19063 leader_election.cc:302] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef2edaf9eaab479aa94ba298fb06aeec, fb03c1d5fde344aaad142f77f76dd585; no voters:
I20260619 20:58:52.398051 19457 raft_consensus.cc:2781] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:52.398176 19457 raft_consensus.cc:686] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [term 1 LEADER]: Becoming Leader. State: Replica: fb03c1d5fde344aaad142f77f76dd585, State: Running, Role: LEADER
I20260619 20:58:52.398286 19457 consensus_queue.cc:237] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260619 20:58:52.399044 18985 catalog_manager.cc:5495] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb03c1d5fde344aaad142f77f76dd585 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb03c1d5fde344aaad142f77f76dd585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb03c1d5fde344aaad142f77f76dd585" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } health_report { overall_health: HEALTHY } } }
I20260619 20:58:52.458479 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.456046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.98.1:41577)
0619 20:58:52.456110 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:52.456112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:52.456122 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:52.456162 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:52.456163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:52.456185 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:52.456270 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:52.456409 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.456889 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.457670 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:52.457886 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:52.457960 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:52.458096 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:52.458194 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:52.458354 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:52.458386 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:58:52.458397 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:58:52.460296 19258 raft_consensus.cc:1261] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 1 FOLLOWER]: Refusing update from remote peer fb03c1d5fde344aaad142f77f76dd585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:52.460314 19389 raft_consensus.cc:1261] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 1 FOLLOWER]: Refusing update from remote peer fb03c1d5fde344aaad142f77f76dd585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:52.460661 19460 consensus_queue.cc:1035] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2edaf9eaab479aa94ba298fb06aeec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:52.460804 19457 consensus_queue.cc:1035] T 11af2518d022499489ea8da291505713 P fb03c1d5fde344aaad142f77f76dd585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fdde4821bd4e969ea5ed5f8a5f8dd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:52.463681 19465 mvcc.cc:205] Tried to move back new op lower bound from 7298673592154877952 to 7298673591904006144. Current Snapshot: MvccSnapshot[applied={T|T < 7298673592154877952 or (T in {7298673592154877952})}]
I20260619 20:58:52.463806 19464 mvcc.cc:205] Tried to move back new op lower bound from 7298673592154877952 to 7298673591904006144. Current Snapshot: MvccSnapshot[applied={T|T < 7298673592154877952}]
I20260619 20:58:52.463117 19463 mvcc.cc:205] Tried to move back new op lower bound from 7298673592154877952 to 7298673591904006144. Current Snapshot: MvccSnapshot[applied={T|T < 7298673592154877952}]
I20260619 20:58:52.466003 18985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:52.466091 18985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:52.466811 18985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b5f618168b4f34942efe028c7dfd06: Sending DeleteTablet for 3 replicas of tablet 11af2518d022499489ea8da291505713
I20260619 20:58:52.467139 19369 tablet_service.cc:1514] Processing DeleteTablet for tablet 11af2518d022499489ea8da291505713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:52 UTC) from {username='slave'} at 127.0.0.1:48418
I20260619 20:58:52.467222 19238 tablet_service.cc:1514] Processing DeleteTablet for tablet 11af2518d022499489ea8da291505713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:52 UTC) from {username='slave'} at 127.0.0.1:45068
I20260619 20:58:52.467322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19045
I20260619 20:58:52.467397 19107 tablet_service.cc:1514] Processing DeleteTablet for tablet 11af2518d022499489ea8da291505713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:52 UTC) from {username='slave'} at 127.0.0.1:41042
I20260619 20:58:52.467901 19473 tablet_replica.cc:330] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: stopping tablet replica
I20260619 20:58:52.467994 19473 raft_consensus.cc:2229] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:52.468145 19473 raft_consensus.cc:2258] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:52.468945 19474 tablet_replica.cc:330] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: stopping tablet replica
I20260619 20:58:52.469046 19474 raft_consensus.cc:2229] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:52.468974 19473 ts_tablet_manager.cc:1905] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:52.469157 19474 raft_consensus.cc:2258] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:52.469561 19474 ts_tablet_manager.cc:1905] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:52.471024 19474 ts_tablet_manager.cc:1918] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:52.471082 19473 ts_tablet_manager.cc:1918] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:52.471110 19474 log.cc:1196] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/11af2518d022499489ea8da291505713
I20260619 20:58:52.471154 19473 log.cc:1196] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/11af2518d022499489ea8da291505713
I20260619 20:58:52.471380 19474 ts_tablet_manager.cc:1939] T 11af2518d022499489ea8da291505713 P ef2edaf9eaab479aa94ba298fb06aeec: Deleting consensus metadata
I20260619 20:58:52.471414 19473 ts_tablet_manager.cc:1939] T 11af2518d022499489ea8da291505713 P e3fdde4821bd4e969ea5ed5f8a5f8dd4: Deleting consensus metadata
I20260619 20:58:52.471809 18971 catalog_manager.cc:4841] TS e3fdde4821bd4e969ea5ed5f8a5f8dd4 (127.0.98.3:33491): tablet 11af2518d022499489ea8da291505713 (table test-table [id=0c220e573488439397252d745c690da5]) successfully deleted
W20260619 20:58:52.473315 18969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41577 (error 108)
I20260619 20:58:52.473570 18972 catalog_manager.cc:4841] TS ef2edaf9eaab479aa94ba298fb06aeec (127.0.98.2:38311): tablet 11af2518d022499489ea8da291505713 (table test-table [id=0c220e573488439397252d745c690da5]) successfully deleted
I20260619 20:58:52.473841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19176
I20260619 20:58:52.473913 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:52.473760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41577 (local address 127.0.0.1:41060)
0619 20:58:52.473826 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:52.473865 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 20:58:52.475390 18969 catalog_manager.cc:4584] TS fb03c1d5fde344aaad142f77f76dd585 (127.0.98.1:41577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11af2518d022499489ea8da291505713: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41577: connect: Connection refused (error 111)
I20260619 20:58:52.478899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19307
I20260619 20:58:52.483358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 18954
2026-06-19T20:58:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (955 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:52 dist-test-slave-8bgz krb5kdc[19478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:52 dist-test-slave-8bgz krb5kdc[19478](info): set up 2 sockets
Jun 19 20:58:52 dist-test-slave-8bgz krb5kdc[19478](info): commencing operation
krb5kdc: starting...
W20260619 20:58:53.518290 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:53 dist-test-slave-8bgz krb5kdc[19478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:58:53.629046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:53.707988 19491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:53.708202 19491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:53.708248 19491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:53.709313 19491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:53.709376 19491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:53.709400 19491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:53.709420 19491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:53.709458 19491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:53.710856 19491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:53.711141 19491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:53.711349 19491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:53.716012 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:53.716084 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:53.716040 19491 server_base.cc:1034] running on GCE node
W20260619 20:58:53.716012 19496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:53.716599 19491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:53.716879 19491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:53.718042 19491 hybrid_clock.cc:648] HybridClock initialized: now 1781902733718038 us; error 36 us; skew 500 ppm
Jun 19 20:58:53 dist-test-slave-8bgz krb5kdc[19478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:53.720395 19491 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:58:53.720851 19491 webserver.cc:460] Webserver started at http://127.0.98.62:44245/ using document root <none> and password file <none>
I20260619 20:58:53.721065 19491 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:53.721130 19491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:53.721273 19491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:53.722326 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "4187cff359934e57970918c8799f36f6"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "303037b7e91db2e53c995d853d406aef"
server_key_iv: "9bad4b8475f567418f3d91d673ff26a6"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.722735 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "4187cff359934e57970918c8799f36f6"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "303037b7e91db2e53c995d853d406aef"
server_key_iv: "9bad4b8475f567418f3d91d673ff26a6"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.724017 19491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:58:53.724735 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:53.724903 19491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:53.724994 19491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "4187cff359934e57970918c8799f36f6"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "303037b7e91db2e53c995d853d406aef"
server_key_iv: "9bad4b8475f567418f3d91d673ff26a6"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.725075 19491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:53.742645 19491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:53.744585 19491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:53.744742 19491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:53.748744 19491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37003
I20260619 20:58:53.748780 19558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37003 every 8 connection(s)
I20260619 20:58:53.749142 19491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:53.749958 19559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:53.752254 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: Bootstrap starting.
I20260619 20:58:53.752830 19559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:53.753122 19559 log.cc:824] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:53.753198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19491
I20260619 20:58:53.753355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:53.753577 392 external_mini_cluster.cc:1428] Setting key 1a1a1d9dc33798cf16b377af176a40c5
I20260619 20:58:53.753939 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: No bootstrap required, opened a new log
I20260619 20:58:53.755648 19559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } }
I20260619 20:58:53.755779 19559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:53.755801 19559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4187cff359934e57970918c8799f36f6, State: Initialized, Role: FOLLOWER
I20260619 20:58:53.755956 19559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } }
I20260619 20:58:53.756063 19559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:53.756096 19559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:53.756150 19559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:53.757109 19559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } }
I20260619 20:58:53.757195 19559 leader_election.cc:302] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4187cff359934e57970918c8799f36f6; no voters:
I20260619 20:58:53.757377 19559 leader_election.cc:288] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:53.757424 19564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:53.757625 19564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [term 1 LEADER]: Becoming Leader. State: Replica: 4187cff359934e57970918c8799f36f6, State: Running, Role: LEADER
I20260619 20:58:53.757627 19559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:53.757778 19564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } }
I20260619 20:58:53.758195 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4187cff359934e57970918c8799f36f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } } }
I20260619 20:58:53.758315 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:53.758545 19566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4187cff359934e57970918c8799f36f6. Latest consensus state: current_term: 1 leader_uuid: "4187cff359934e57970918c8799f36f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4187cff359934e57970918c8799f36f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37003 } } }
I20260619 20:58:53.758875 19566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6 [sys.catalog]: This master's current role is: LEADER
W20260619 20:58:53.759435 19580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:58:53.762529 19580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:58:53.762629 19572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 20:58:53 dist-test-slave-8bgz krb5kdc[19478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:53.762801 19572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:53.764283 19572 catalog_manager.cc:1347] Generated new cluster ID: f9984f3d9632444baee12bd9ae4b571e
I20260619 20:58:53.764328 19572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:53.766247 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:53.754076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.0.98.62:37003)
0619 20:58:53.754276 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:53.754287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:53.754318 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:58:53.755658 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:53.755665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:53.755716 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:53.755901 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:53.756706 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.757895 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.759085 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.759686 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.759798 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:53.763156 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:53.763160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:53.763160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:53.763171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:53.764776 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:53.765146 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:53.765151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:53.765151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:53.765254 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:53.765464 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:53.765465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:53.765465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:53.765675 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:53.765834 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:53.765904 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:58:53.766031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260619 20:58:53.785386 19572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:53.785912 19572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:53.790478 19572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: Generated new TSK 0
I20260619 20:58:53.790685 19572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:58:53.819132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:53.913199 19585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:53.913393 19585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:53.913414 19585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:53.914543 19585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:53.914587 19585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:53.914615 19585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:53.916061 19585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:53.916282 19585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:53.916515 19585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:53.921221 19591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:53.921193 19593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:53.921203 19590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:53.921469 19585 server_base.cc:1034] running on GCE node
I20260619 20:58:53.921700 19585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:53.921926 19585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:53.923098 19585 hybrid_clock.cc:648] HybridClock initialized: now 1781902733923081 us; error 36 us; skew 500 ppm
Jun 19 20:58:53 dist-test-slave-8bgz krb5kdc[19478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:53.925531 19585 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:58:53.925980 19585 webserver.cc:460] Webserver started at http://127.0.98.1:37537/ using document root <none> and password file <none>
I20260619 20:58:53.926198 19585 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:53.926312 19585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:53.926447 19585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:53.927414 19585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c54fe3aefc8410591e24c2d392d5e47"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "8f8df9c247aa9f652a03a79f5db393a2"
server_key_iv: "0c688a9b2c51db1e2bba630040e5bd7e"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.927750 19585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c54fe3aefc8410591e24c2d392d5e47"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "8f8df9c247aa9f652a03a79f5db393a2"
server_key_iv: "0c688a9b2c51db1e2bba630040e5bd7e"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.929019 19585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:53.929761 19600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:53.930006 19585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:53.930110 19585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "4c54fe3aefc8410591e24c2d392d5e47"
format_stamp: "Formatted at 2026-06-19 20:58:53 on dist-test-slave-8bgz"
server_key: "8f8df9c247aa9f652a03a79f5db393a2"
server_key_iv: "0c688a9b2c51db1e2bba630040e5bd7e"
server_key_version: "encryptionkey@0"
I20260619 20:58:53.930186 19585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:53.952641 19585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:53.954479 19585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:53.954612 19585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:53.954841 19585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:53.955137 19585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:53.955193 19585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:53.955235 19585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:53.955260 19585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:53.961340 19585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40563
I20260619 20:58:53.961385 19713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40563 every 8 connection(s)
I20260619 20:58:53.961660 19585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:53.964056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19585
I20260619 20:58:53.964139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:53.964327 392 external_mini_cluster.cc:1428] Setting key a5a7d3e86d80b54f00298db57799b988
Jun 19 20:58:53 dist-test-slave-8bgz krb5kdc[19478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:58:53.971508 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:53.962508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52915 (local address 127.0.98.62:37003)
0619 20:58:53.962561 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:53.962564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:53.962701 (+ 137us) server_negotiation.cc:407] Connection header received
0619 20:58:53.963576 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:53.963577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:53.963589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:53.963633 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:53.964285 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.965044 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.966061 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.966354 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.966437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:53.968583 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:53.968585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:53.968585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:53.968593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:53.969976 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:53.970513 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:53.970515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:53.970515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:53.970610 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:53.970860 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:53.970861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:53.970862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:53.970979 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:53.971060 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:53.971330 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 20:58:53.971418 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:58:53.971693 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:53.962228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37003 (local address 127.0.98.1:52915)
0619 20:58:53.962589 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:53.962623 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:53.963497 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:53.963689 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:53.963692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:53.963929 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:53.964133 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:53.964139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.965154 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.965156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:53.965967 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:53.965969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:53.966449 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:53.966450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:53.966583 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:53.967170 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:53.967178 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:53.968488 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:53.970083 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:53.970090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:53.970092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:53.970422 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:53.970678 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:53.970679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:53.970679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:53.970791 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:53.971053 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:53.971059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:53.971194 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 20:58:53.971294 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 20:58:53.971440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":111,"threads_started":1}
I20260619 20:58:53.972148 19714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37003
I20260619 20:58:53.972226 19714 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:53.972404 19714 heartbeater.cc:511] Master 127.0.98.62:37003 requested a full tablet report, sending...
I20260619 20:58:53.972930 19523 ts_manager.cc:194] Registered new tserver with Master: 4c54fe3aefc8410591e24c2d392d5e47 (127.0.98.1:40563)
I20260619 20:58:53.973626 19523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:58:53.988579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:54.066637 19719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:54.066836 19719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:54.066857 19719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:54.068040 19719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:54.068081 19719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:54.068107 19719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:54.069592 19719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:54.069823 19719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:54.070041 19719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:54.074847 19727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.074859 19725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.074836 19724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.075095 19719 server_base.cc:1034] running on GCE node
I20260619 20:58:54.075316 19719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:54.075538 19719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:54.076674 19719 hybrid_clock.cc:648] HybridClock initialized: now 1781902734076663 us; error 28 us; skew 500 ppm
Jun 19 20:58:54 dist-test-slave-8bgz krb5kdc[19478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:54.079068 19719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:58:54.079530 19719 webserver.cc:460] Webserver started at http://127.0.98.2:38293/ using document root <none> and password file <none>
I20260619 20:58:54.079746 19719 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:54.079788 19719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:54.079932 19719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:54.080891 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "45291ca7bde24b0894ba35a75dc1fa20"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "91277acf177502caa86f7bf3c71f5982"
server_key_iv: "cb4b370e9b72bd7de45857be9c42b0c3"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.081243 19719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45291ca7bde24b0894ba35a75dc1fa20"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "91277acf177502caa86f7bf3c71f5982"
server_key_iv: "cb4b370e9b72bd7de45857be9c42b0c3"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.082531 19719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:54.083199 19734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.083377 19719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:54.083457 19719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "45291ca7bde24b0894ba35a75dc1fa20"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "91277acf177502caa86f7bf3c71f5982"
server_key_iv: "cb4b370e9b72bd7de45857be9c42b0c3"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.083542 19719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:54.112375 19719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:54.114295 19719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:54.114449 19719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:54.114679 19719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:54.114976 19719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:54.115006 19719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.115028 19719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:54.115074 19719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.120805 19719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44685
I20260619 20:58:54.120848 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44685 every 8 connection(s)
I20260619 20:58:54.121145 19719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:54.122922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19719
I20260619 20:58:54.123059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:54.123274 392 external_mini_cluster.cc:1428] Setting key bb0d50e53d5f28e0824551d9ed3573a8
Jun 19 20:58:54 dist-test-slave-8bgz krb5kdc[19478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:54.131623 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.121839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37839 (local address 127.0.98.62:37003)
0619 20:58:54.121900 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.121902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.122100 (+ 198us) server_negotiation.cc:407] Connection header received
0619 20:58:54.123254 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.123255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.123266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.123350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:54.124134 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.124623 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.125882 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.126200 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.126290 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.128595 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:54.128597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:54.128598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:54.128606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:54.130294 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.130664 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.130666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.130666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.130735 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.130963 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.130965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.130965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.131109 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:54.131182 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.131413 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.131509 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:58:54.131629 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.121822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37003 (local address 127.0.98.2:37839)
0619 20:58:54.121975 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.122015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.123143 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.123425 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.123430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.123769 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:54.124043 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.124050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.124712 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.124713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.125796 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.125799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.126307 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.126308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.126419 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.126908 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:54.126915 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:54.128503 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:54.130370 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:54.130376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:54.130377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:54.130587 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:54.130803 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:54.130803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:54.130803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:54.130896 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:54.131176 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:54.131179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:54.131280 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.131361 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.131486 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:58:54.132273 19848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37003
I20260619 20:58:54.132347 19848 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:54.132530 19848 heartbeater.cc:511] Master 127.0.98.62:37003 requested a full tablet report, sending...
I20260619 20:58:54.132966 19523 ts_manager.cc:194] Registered new tserver with Master: 45291ca7bde24b0894ba35a75dc1fa20 (127.0.98.2:44685)
I20260619 20:58:54.133335 19523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:58:54.147161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:54.225188 19853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:54.225395 19853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:54.225435 19853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:54.226644 19853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:54.226704 19853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:54.226750 19853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:54.228278 19853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:54.228521 19853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:54.228739 19853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:54.233677 19859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.233668 19853 server_base.cc:1034] running on GCE node
W20260619 20:58:54.233624 19861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.233642 19858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.234047 19853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:54.234300 19853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:54.235474 19853 hybrid_clock.cc:648] HybridClock initialized: now 1781902734235460 us; error 36 us; skew 500 ppm
Jun 19 20:58:54 dist-test-slave-8bgz krb5kdc[19478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:54.237902 19853 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:58:54.238355 19853 webserver.cc:460] Webserver started at http://127.0.98.3:38423/ using document root <none> and password file <none>
I20260619 20:58:54.238580 19853 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:54.238660 19853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:54.238797 19853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:54.239774 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6bb0dfff70d416db6345c1f2e805a45"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "68574c912c06809ad535c2f80d8c961f"
server_key_iv: "7edf0a9c271da5bab83e88ff5a885022"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.240115 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6bb0dfff70d416db6345c1f2e805a45"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "68574c912c06809ad535c2f80d8c961f"
server_key_iv: "7edf0a9c271da5bab83e88ff5a885022"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.241439 19853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:58:54.242100 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.242313 19853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:58:54.242398 19853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "f6bb0dfff70d416db6345c1f2e805a45"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "68574c912c06809ad535c2f80d8c961f"
server_key_iv: "7edf0a9c271da5bab83e88ff5a885022"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.242479 19853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:54.279335 19853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:54.281698 19853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:54.281863 19853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:54.282100 19853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:54.282469 19853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:54.282528 19853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.282569 19853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:54.282596 19853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.288146 19853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44247
I20260619 20:58:54.288234 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44247 every 8 connection(s)
I20260619 20:58:54.288563 19853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:54.291503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 19853
I20260619 20:58:54.291618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:54.291806 392 external_mini_cluster.cc:1428] Setting key 427d66bb062caab0ff1fe8d227a6bc35
Jun 19 20:58:54 dist-test-slave-8bgz krb5kdc[19478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:58:54.298458 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.289270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55401 (local address 127.0.98.62:37003)
0619 20:58:54.289347 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.289350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.289537 (+ 187us) server_negotiation.cc:407] Connection header received
0619 20:58:54.290570 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.290572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.290586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.290661 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:54.291318 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.292064 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.293302 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.293486 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.293548 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.295610 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:54.295612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:54.295612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:54.295619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:54.297027 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.297468 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.297469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.297469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.297533 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.297780 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.297781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.297781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.297896 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:54.297986 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.298296 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.298370 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:58:54.298638 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.289242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37003 (local address 127.0.98.3:55401)
0619 20:58:54.289422 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.289456 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.290441 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.290753 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.290756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.290994 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:54.291194 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.291199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.292252 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.292253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.293235 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.293238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.293569 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.293570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.293666 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.294073 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:58:54.294081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:54.295512 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:54.297134 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:54.297143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:54.297145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:54.297369 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:54.297598 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:58:54.297599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:58:54.297599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:58:54.297709 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:58:54.297979 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:54.297984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:54.298126 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.298269 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.298417 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260619 20:58:54.299131 19982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37003
I20260619 20:58:54.299233 19982 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:54.299365 19982 heartbeater.cc:511] Master 127.0.98.62:37003 requested a full tablet report, sending...
I20260619 20:58:54.299722 19523 ts_manager.cc:194] Registered new tserver with Master: f6bb0dfff70d416db6345c1f2e805a45 (127.0.98.3:44247)
I20260619 20:58:54.300051 19523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55401
I20260619 20:58:54.303556 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:54.309996 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.304774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.0.98.62:37003)
0619 20:58:54.304832 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.304834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.304887 (+ 53us) server_negotiation.cc:407] Connection header received
0619 20:58:54.304947 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.304948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.304955 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.305002 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:54.305489 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.305931 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.306696 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.306863 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.306914 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.307871 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:54.307872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:54.307873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:54.307878 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:54.309152 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.309420 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.309420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.309420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.309465 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:54.309622 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:54.309623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:54.309623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:54.309738 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:54.309806 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.309891 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.309941 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:58:54.311378 19523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:54.312872 19523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:54.319644 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44247 (local address 127.0.0.1:39568)
0619 20:58:54.315559 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.315599 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.315671 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.316571 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.316572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.316574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:54.316614 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.316615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317738 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.317739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.318559 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.318561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319342 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.319419 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319431 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.319503 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.319556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":62,"threads_started":1}
I20260619 20:58:54.319658 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.98.3:44247)
0619 20:58:54.316408 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.316412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.316422 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:54.316457 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.316459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.316474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.316562 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:54.316682 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317626 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.318650 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319250 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319332 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319385 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.319532 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.319571 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":77,"threads_started":1}
I20260619 20:58:54.319831 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.98.2:44685)
0619 20:58:54.315817 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.315822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.315835 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:58:54.315880 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.315883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.315900 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.316030 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:54.316171 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317316 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.318665 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319290 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319379 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319435 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.319689 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.319740 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260619 20:58:54.320184 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40563 (local address 127.0.0.1:39118)
0619 20:58:54.315633 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.315642 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.315682 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.316270 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.316271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.316273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:54.316309 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.316311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317277 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.317278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.318918 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.318920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319839 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.319973 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319999 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.320057 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.320102 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":5,"thread_start_us":144,"threads_started":1}
I20260619 20:58:54.320221 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.98.1:40563)
0619 20:58:54.316109 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.316114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.316123 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:58:54.316164 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.316166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.316181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.316262 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:54.316393 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317178 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319003 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319747 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319836 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319890 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.320095 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.320144 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260619 20:58:54.319974 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.315331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44685 (local address 127.0.0.1:59116)
0619 20:58:54.315782 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.315789 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.315831 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.316037 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.316040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.316042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:54.316102 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.316103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.317420 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.317421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.318572 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.318575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.319466 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.319467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.319545 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.319549 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.319595 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.319631 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":7,"thread_start_us":143,"threads_started":1}
I20260619 20:58:54.320746 19916 tablet_service.cc:1467] Processing CreateTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a (DEFAULT_TABLE table=test-table [id=5eeebda081db4afa8ec276a58ae9a27e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:54.320834 19648 tablet_service.cc:1467] Processing CreateTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a (DEFAULT_TABLE table=test-table [id=5eeebda081db4afa8ec276a58ae9a27e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:54.321126 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d45bd7cb7f43fa90f69fcb6110645a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:54.321134 19648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d45bd7cb7f43fa90f69fcb6110645a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:54.322561 19782 tablet_service.cc:1467] Processing CreateTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a (DEFAULT_TABLE table=test-table [id=5eeebda081db4afa8ec276a58ae9a27e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:54.322847 19782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d45bd7cb7f43fa90f69fcb6110645a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:54.324579 19999 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Bootstrap starting.
I20260619 20:58:54.325282 19999 tablet_bootstrap.cc:654] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:54.325848 20001 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Bootstrap starting.
I20260619 20:58:54.325898 19999 log.cc:824] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:54.326195 20000 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Bootstrap starting.
I20260619 20:58:54.326586 20001 tablet_bootstrap.cc:654] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:54.326773 19999 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: No bootstrap required, opened a new log
I20260619 20:58:54.326844 19999 ts_tablet_manager.cc:1397] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:58:54.326900 20000 tablet_bootstrap.cc:654] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:54.327178 20001 log.cc:824] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:54.327466 20000 log.cc:824] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:54.327982 20001 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: No bootstrap required, opened a new log
I20260619 20:58:54.328056 20001 ts_tablet_manager.cc:1397] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:58:54.328181 20000 tablet_bootstrap.cc:492] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: No bootstrap required, opened a new log
I20260619 20:58:54.328249 20000 ts_tablet_manager.cc:1397] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:54.328899 19999 raft_consensus.cc:348] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.329059 19999 raft_consensus.cc:374] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:54.329092 19999 raft_consensus.cc:729] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c54fe3aefc8410591e24c2d392d5e47, State: Initialized, Role: FOLLOWER
I20260619 20:58:54.329214 19999 consensus_queue.cc:260] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.329481 19999 ts_tablet_manager.cc:1428] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:54.329608 19714 heartbeater.cc:503] Master 127.0.98.62:37003 was elected leader, sending a full tablet report...
I20260619 20:58:54.330006 20000 raft_consensus.cc:348] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.330178 20000 raft_consensus.cc:374] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:54.330215 20000 raft_consensus.cc:729] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bb0dfff70d416db6345c1f2e805a45, State: Initialized, Role: FOLLOWER
I20260619 20:58:54.330178 20001 raft_consensus.cc:348] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.330365 20001 raft_consensus.cc:374] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:54.330338 20000 consensus_queue.cc:260] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.330400 20001 raft_consensus.cc:729] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45291ca7bde24b0894ba35a75dc1fa20, State: Initialized, Role: FOLLOWER
I20260619 20:58:54.330526 20001 consensus_queue.cc:260] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.330636 20000 ts_tablet_manager.cc:1428] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:58:54.330783 20001 ts_tablet_manager.cc:1428] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:54.330863 19982 heartbeater.cc:503] Master 127.0.98.62:37003 was elected leader, sending a full tablet report...
I20260619 20:58:54.330896 19848 heartbeater.cc:503] Master 127.0.98.62:37003 was elected leader, sending a full tablet report...
W20260619 20:58:54.371675 19849 tablet.cc:2369] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:58:54.462298 19715 tablet.cc:2369] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:54.524297 20007 raft_consensus.cc:482] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:54.524415 20007 raft_consensus.cc:504] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.524808 20007 leader_election.cc:288] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c54fe3aefc8410591e24c2d392d5e47 (127.0.98.1:40563), f6bb0dfff70d416db6345c1f2e805a45 (127.0.98.3:44247)
I20260619 20:58:54.528672 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.525092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33529 (local address 127.0.98.3:44247)
0619 20:58:54.525139 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.525141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.525150 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:58:54.525177 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.525178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.525186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.525266 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:54.525469 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.526255 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.527750 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.528319 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.528389 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.528398 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.528570 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.528607 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:58:54.528671 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.525058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39451 (local address 127.0.98.1:40563)
0619 20:58:54.525107 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.525109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.525246 (+ 137us) server_negotiation.cc:407] Connection header received
0619 20:58:54.525308 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.525309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.525328 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.525369 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:54.525558 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.526257 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.527788 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.528320 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.528389 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.528398 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.528570 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.528606 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:58:54.528688 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.524989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44247 (local address 127.0.98.2:33529)
0619 20:58:54.525047 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.525059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.525102 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.525286 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.525287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.525290 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:54.525387 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.525390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.526343 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.526343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.527658 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.527661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.528396 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.528397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.528489 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.528494 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.528553 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.528599 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":45}
I20260619 20:58:54.528694 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.525001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40563 (local address 127.0.98.2:39451)
0619 20:58:54.525171 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.525180 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.525235 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.525391 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.525392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.525393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:54.525487 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.525490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.526338 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.526338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.527658 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.527661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.528397 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.528397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.528504 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.528507 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.528560 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.528598 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":56,"thread_start_us":101,"threads_started":1}
I20260619 20:58:54.529104 19668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d45bd7cb7f43fa90f69fcb6110645a" candidate_uuid: "45291ca7bde24b0894ba35a75dc1fa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c54fe3aefc8410591e24c2d392d5e47" is_pre_election: true
I20260619 20:58:54.529160 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d45bd7cb7f43fa90f69fcb6110645a" candidate_uuid: "45291ca7bde24b0894ba35a75dc1fa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bb0dfff70d416db6345c1f2e805a45" is_pre_election: true
I20260619 20:58:54.529263 19936 raft_consensus.cc:2454] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45291ca7bde24b0894ba35a75dc1fa20 in term 0.
I20260619 20:58:54.529261 19668 raft_consensus.cc:2454] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45291ca7bde24b0894ba35a75dc1fa20 in term 0.
I20260619 20:58:54.529460 19738 leader_election.cc:302] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45291ca7bde24b0894ba35a75dc1fa20, 4c54fe3aefc8410591e24c2d392d5e47; no voters:
I20260619 20:58:54.529633 20007 raft_consensus.cc:2781] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:54.529707 20007 raft_consensus.cc:482] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:54.529735 20007 raft_consensus.cc:3037] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:54.530506 20007 raft_consensus.cc:504] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.530678 20007 leader_election.cc:288] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [CANDIDATE]: Term 1 election: Requested vote from peers 4c54fe3aefc8410591e24c2d392d5e47 (127.0.98.1:40563), f6bb0dfff70d416db6345c1f2e805a45 (127.0.98.3:44247)
I20260619 20:58:54.530849 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d45bd7cb7f43fa90f69fcb6110645a" candidate_uuid: "45291ca7bde24b0894ba35a75dc1fa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bb0dfff70d416db6345c1f2e805a45"
I20260619 20:58:54.530901 19668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d45bd7cb7f43fa90f69fcb6110645a" candidate_uuid: "45291ca7bde24b0894ba35a75dc1fa20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c54fe3aefc8410591e24c2d392d5e47"
I20260619 20:58:54.530964 19936 raft_consensus.cc:3037] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:54.530965 19668 raft_consensus.cc:3037] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:54.531659 19936 raft_consensus.cc:2454] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45291ca7bde24b0894ba35a75dc1fa20 in term 1.
I20260619 20:58:54.531664 19668 raft_consensus.cc:2454] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45291ca7bde24b0894ba35a75dc1fa20 in term 1.
I20260619 20:58:54.531852 19738 leader_election.cc:302] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45291ca7bde24b0894ba35a75dc1fa20, 4c54fe3aefc8410591e24c2d392d5e47; no voters:
I20260619 20:58:54.531970 20007 raft_consensus.cc:2781] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:54.532104 20007 raft_consensus.cc:686] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 1 LEADER]: Becoming Leader. State: Replica: 45291ca7bde24b0894ba35a75dc1fa20, State: Running, Role: LEADER
I20260619 20:58:54.532225 20007 consensus_queue.cc:237] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } }
I20260619 20:58:54.533053 19522 catalog_manager.cc:5495] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45291ca7bde24b0894ba35a75dc1fa20 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "45291ca7bde24b0894ba35a75dc1fa20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45291ca7bde24b0894ba35a75dc1fa20" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } health_report { overall_health: UNKNOWN } } }
W20260619 20:58:54.539070 19983 tablet.cc:2369] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:58:54.546803 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.544237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.0.98.2:44685)
0619 20:58:54.544299 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.544302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.544312 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:58:54.544409 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.544410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.544428 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.544496 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:54.544634 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.545182 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.545955 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.546248 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.546305 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.546459 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:54.546565 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:54.546654 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.546695 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.546714 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:58:54.548450 19936 raft_consensus.cc:1261] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 1 FOLLOWER]: Refusing update from remote peer 45291ca7bde24b0894ba35a75dc1fa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:54.548492 19668 raft_consensus.cc:1261] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 1 FOLLOWER]: Refusing update from remote peer 45291ca7bde24b0894ba35a75dc1fa20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:54.548844 20009 consensus_queue.cc:1035] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bb0dfff70d416db6345c1f2e805a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:54.548964 20007 consensus_queue.cc:1035] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c54fe3aefc8410591e24c2d392d5e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:54.551110 20012 mvcc.cc:205] Tried to move back new op lower bound from 7298673600707649536 to 7298673600644509696. Current Snapshot: MvccSnapshot[applied={T|T < 7298673600707649536}]
I20260619 20:58:54.551266 20014 mvcc.cc:205] Tried to move back new op lower bound from 7298673600707649536 to 7298673600644509696. Current Snapshot: MvccSnapshot[applied={T|T < 7298673600707649536}]
I20260619 20:58:54.553552 19523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:54.553637 19523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:54.554355 19523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4187cff359934e57970918c8799f36f6: Sending DeleteTablet for 3 replicas of tablet c2d45bd7cb7f43fa90f69fcb6110645a
I20260619 20:58:54.554641 19648 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39118
I20260619 20:58:54.554668 19782 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59116
I20260619 20:58:54.554894 20021 tablet_replica.cc:330] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: stopping tablet replica
I20260619 20:58:54.555033 20021 raft_consensus.cc:2229] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:54.555047 20022 tablet_replica.cc:330] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: stopping tablet replica
I20260619 20:58:54.555058 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d45bd7cb7f43fa90f69fcb6110645a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39568
I20260619 20:58:54.555125 20022 raft_consensus.cc:2229] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:54.555172 20021 raft_consensus.cc:2258] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:54.555199 20022 raft_consensus.cc:2258] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:54.555399 20023 tablet_replica.cc:330] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: stopping tablet replica
I20260619 20:58:54.555480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19585
I20260619 20:58:54.555526 20023 raft_consensus.cc:2229] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:54.555599 20021 ts_tablet_manager.cc:1905] T c2d45bd7cb7f43fa90f69fcb6110645a P 4c54fe3aefc8410591e24c2d392d5e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:54.555613 20023 raft_consensus.cc:2258] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:54.557128 20023 ts_tablet_manager.cc:1905] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:54.558880 20023 ts_tablet_manager.cc:1918] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:54.558955 20023 log.cc:1196] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/c2d45bd7cb7f43fa90f69fcb6110645a
I20260619 20:58:54.559319 20023 ts_tablet_manager.cc:1939] T c2d45bd7cb7f43fa90f69fcb6110645a P f6bb0dfff70d416db6345c1f2e805a45: Deleting consensus metadata
I20260619 20:58:54.559715 19509 catalog_manager.cc:4841] TS f6bb0dfff70d416db6345c1f2e805a45 (127.0.98.3:44247): tablet c2d45bd7cb7f43fa90f69fcb6110645a (table test-table [id=5eeebda081db4afa8ec276a58ae9a27e]) successfully deleted
W20260619 20:58:54.561642 19510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40563 (error 108)
I20260619 20:58:54.562140 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.562005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40563 (local address 127.0.0.1:54100)
0619 20:58:54.562064 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.562108 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:58:54.562184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19719
W20260619 20:58:54.562284 19510 catalog_manager.cc:4584] TS 4c54fe3aefc8410591e24c2d392d5e47 (127.0.98.1:40563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2d45bd7cb7f43fa90f69fcb6110645a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40563: connect: Connection refused (error 111)
I20260619 20:58:54.562639 20022 ts_tablet_manager.cc:1905] T c2d45bd7cb7f43fa90f69fcb6110645a P 45291ca7bde24b0894ba35a75dc1fa20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:54.566906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19853
I20260619 20:58:54.566967 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.566854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44685 (local address 127.0.0.1:50002)
0619 20:58:54.566910 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.566944 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:58:54.571693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 19491
2026-06-19T20:58:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T20:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:54Z Disabled control of system clock
I20260619 20:58:54.598642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37861
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:54.677770 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:54.677971 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:54.677992 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:54.679204 20028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:54.679247 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:54.679260 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:54.679273 20028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:54.679286 20028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:54.680617 20028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:54.680861 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:54.681151 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:54.685966 20033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.685985 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.686002 20034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.686069 20028 server_base.cc:1034] running on GCE node
I20260619 20:58:54.686463 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:54.686777 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:54.687935 20028 hybrid_clock.cc:648] HybridClock initialized: now 1781902734687928 us; error 36 us; skew 500 ppm
I20260619 20:58:54.689180 20028 webserver.cc:460] Webserver started at http://127.0.98.62:43101/ using document root <none> and password file <none>
I20260619 20:58:54.689415 20028 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:54.689487 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:54.689643 20028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:54.690722 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "654b01629909499e88192fbb835da6d2"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "f21607d0120db84180afb0e99d0e6c90"
server_key_iv: "2512aa9b1505bc6091242aa2b2836a40"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.691071 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "654b01629909499e88192fbb835da6d2"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "f21607d0120db84180afb0e99d0e6c90"
server_key_iv: "2512aa9b1505bc6091242aa2b2836a40"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.692415 20028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:54.693189 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.693403 20028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:54.693531 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "654b01629909499e88192fbb835da6d2"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "f21607d0120db84180afb0e99d0e6c90"
server_key_iv: "2512aa9b1505bc6091242aa2b2836a40"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.693609 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:54.700306 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:54.700587 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:54.700723 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:54.704538 20028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37861
I20260619 20:58:54.704567 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37861 every 8 connection(s)
I20260619 20:58:54.704887 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:54.705791 20095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:54.708015 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: Bootstrap starting.
I20260619 20:58:54.708518 20095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:54.708755 20095 log.cc:824] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:54.709331 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: No bootstrap required, opened a new log
I20260619 20:58:54.710959 20095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } }
I20260619 20:58:54.711090 20095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:54.711112 20095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654b01629909499e88192fbb835da6d2, State: Initialized, Role: FOLLOWER
I20260619 20:58:54.711261 20095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } }
I20260619 20:58:54.711334 20095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:54.711356 20095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:54.711383 20095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:54.711825 20095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } }
I20260619 20:58:54.711905 20095 leader_election.cc:302] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654b01629909499e88192fbb835da6d2; no voters:
I20260619 20:58:54.712018 20095 leader_election.cc:288] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:54.712086 20098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:54.712262 20098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [term 1 LEADER]: Becoming Leader. State: Replica: 654b01629909499e88192fbb835da6d2, State: Running, Role: LEADER
I20260619 20:58:54.712283 20095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:54.712471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20028
I20260619 20:58:54.712584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:54.712798 392 external_mini_cluster.cc:1428] Setting key d83c2dfa3827926baa859ac3b72446ba
I20260619 20:58:54.712915 20098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } }
I20260619 20:58:54.713502 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654b01629909499e88192fbb835da6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } } }
I20260619 20:58:54.713603 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:54.713806 20100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654b01629909499e88192fbb835da6d2. Latest consensus state: current_term: 1 leader_uuid: "654b01629909499e88192fbb835da6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654b01629909499e88192fbb835da6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37861 } } }
I20260619 20:58:54.713907 20100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2 [sys.catalog]: This master's current role is: LEADER
W20260619 20:58:54.714267 20114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:58:54.714320 20114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:58:54.714403 20115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:54.714589 20115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:54.716154 20115 catalog_manager.cc:1347] Generated new cluster ID: a421efa2649c4b45a8085109bfaf4ad9
I20260619 20:58:54.716248 20115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:54.718693 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.713221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.98.62:37861)
0619 20:58:54.714073 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.714082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.714115 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:58:54.714267 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.714274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.714381 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.714788 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:54.714883 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.715965 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.716792 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.717206 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.717331 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.717990 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:54.717994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:54.717994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:54.718160 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:54.718190 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:54.718237 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:54.718317 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.718407 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.718498 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":74,"threads_started":1}
I20260619 20:58:54.725301 20115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:54.725962 20115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:54.730944 20115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: Generated new TSK 0
I20260619 20:58:54.731146 20115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:54.739254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:54.841408 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:54.841632 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:54.841666 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:54.842847 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:54.842911 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:54.842960 20119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:58:54.844372 20119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:54.844614 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:54.844895 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:54.850685 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.850785 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.850693 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.851018 20119 server_base.cc:1034] running on GCE node
I20260619 20:58:54.851220 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:54.851475 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:54.852664 20119 hybrid_clock.cc:648] HybridClock initialized: now 1781902734852657 us; error 33 us; skew 500 ppm
I20260619 20:58:54.854067 20119 webserver.cc:460] Webserver started at http://127.0.98.1:38607/ using document root <none> and password file <none>
I20260619 20:58:54.854333 20119 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:54.854394 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:54.854527 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:54.855849 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5da22923a87406e803f80e1a8b54f00"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "b667ef8fb0fa063b7d4453bcb7cd58de"
server_key_iv: "fe8d4bb0faf3e23c93d240f6fb66e856"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.856246 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5da22923a87406e803f80e1a8b54f00"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "b667ef8fb0fa063b7d4453bcb7cd58de"
server_key_iv: "fe8d4bb0faf3e23c93d240f6fb66e856"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.857795 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:58:54.858589 20133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.858755 20119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.858839 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal
uuid: "f5da22923a87406e803f80e1a8b54f00"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "b667ef8fb0fa063b7d4453bcb7cd58de"
server_key_iv: "fe8d4bb0faf3e23c93d240f6fb66e856"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.858889 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:54.870520 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:54.870775 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:54.870913 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:54.871140 20119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:54.871440 20119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:54.871496 20119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.871534 20119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:54.871562 20119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.877061 20119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38117
I20260619 20:58:54.877200 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38117 every 8 connection(s)
I20260619 20:58:54.877604 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:58:54.882615 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.878323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39237 (local address 127.0.98.62:37861)
0619 20:58:54.878379 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:54.878381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:54.878574 (+ 193us) server_negotiation.cc:407] Connection header received
0619 20:58:54.879027 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:54.879027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:54.879037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:54.879087 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:54.879702 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.880179 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.881161 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.881332 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.881379 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.882169 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:54.882170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:54.882170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:54.882174 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:54.882189 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:54.882193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:54.882407 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:54.882533 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 20:58:54.882558 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:58:54.882819 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:54.878311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37861 (local address 127.0.98.1:39237)
0619 20:58:54.878481 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:54.878510 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:54.878943 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:54.879134 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:54.879137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:54.879144 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:54.879635 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.879640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.880258 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.880259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.881057 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:54.881059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:54.881406 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:54.881407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:54.881537 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:54.882021 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:54.882022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:54.882047 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:54.882069 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:54.882423 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:54.882430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:54.882438 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:58:54.882517 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:58:54.882593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260619 20:58:54.883419 20247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37861
I20260619 20:58:54.883517 20247 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:54.883661 20247 heartbeater.cc:511] Master 127.0.98.62:37861 requested a full tablet report, sending...
I20260619 20:58:54.884172 20059 ts_manager.cc:194] Registered new tserver with Master: f5da22923a87406e803f80e1a8b54f00 (127.0.98.1:38117)
I20260619 20:58:54.884737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20119
I20260619 20:58:54.884826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:58:54.884979 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39237
I20260619 20:58:54.885016 392 external_mini_cluster.cc:1428] Setting key 9c4dc5a59ad02c11576e79969de772f4
I20260619 20:58:54.885983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:54.964028 20250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:54.964335 20250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:54.964371 20250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:54.965476 20250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:54.965536 20250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:54.965576 20250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:58:54.966892 20250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:54.967145 20250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:54.967419 20250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:54.972213 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.972240 20250 server_base.cc:1034] running on GCE node
W20260619 20:58:54.972203 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:54.972222 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:54.972739 20250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:54.972993 20250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:54.974153 20250 hybrid_clock.cc:648] HybridClock initialized: now 1781902734974136 us; error 33 us; skew 500 ppm
I20260619 20:58:54.975365 20250 webserver.cc:460] Webserver started at http://127.0.98.2:41279/ using document root <none> and password file <none>
I20260619 20:58:54.975598 20250 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:54.975647 20250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:54.975800 20250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:54.976845 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data/instance:
uuid: "c418dab3dfd84b048bdbb6866b3673df"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "2bebb329484c7daa0f5c7aba31516da5"
server_key_iv: "50ebe22ce68458373b1487d55de8b3f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.977183 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c418dab3dfd84b048bdbb6866b3673df"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "2bebb329484c7daa0f5c7aba31516da5"
server_key_iv: "50ebe22ce68458373b1487d55de8b3f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.978539 20250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:54.979223 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.979378 20250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:54.979477 20250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal
uuid: "c418dab3dfd84b048bdbb6866b3673df"
format_stamp: "Formatted at 2026-06-19 20:58:54 on dist-test-slave-8bgz"
server_key: "2bebb329484c7daa0f5c7aba31516da5"
server_key_iv: "50ebe22ce68458373b1487d55de8b3f0"
server_key_version: "encryptionkey@0"
I20260619 20:58:54.979550 20250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:54.994390 20250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:54.994668 20250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:54.994819 20250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:54.995045 20250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:54.995347 20250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:54.995397 20250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:54.995432 20250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:54.995460 20250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:55.001236 20250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43593
I20260619 20:58:55.001283 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43593 every 8 connection(s)
I20260619 20:58:55.001579 20250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:58:55.006608 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.002250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57563 (local address 127.0.98.62:37861)
0619 20:58:55.002300 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.002302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.002509 (+ 207us) server_negotiation.cc:407] Connection header received
0619 20:58:55.003033 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.003034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.003053 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.003111 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:55.003769 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.004262 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.005237 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.005458 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.005539 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.006343 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:55.006344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:55.006345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:55.006349 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:55.006361 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:55.006365 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:55.006408 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.006504 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.006533 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:58:55.006835 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.002187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37861 (local address 127.0.98.2:57563)
0619 20:58:55.002369 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.002408 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.002912 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.003178 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.003181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.003190 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:55.003668 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.003672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.004355 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.004356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.005139 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.005141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.005556 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.005557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.005686 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.006195 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:55.006195 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:55.006235 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:55.006255 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:55.006418 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:55.006422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:55.006425 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.006507 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.006599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260619 20:58:55.007297 20378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37861
I20260619 20:58:55.007398 20378 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:55.007608 20378 heartbeater.cc:511] Master 127.0.98.62:37861 requested a full tablet report, sending...
I20260619 20:58:55.007977 20059 ts_manager.cc:194] Registered new tserver with Master: c418dab3dfd84b048bdbb6866b3673df (127.0.98.2:43593)
I20260619 20:58:55.008347 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57563
I20260619 20:58:55.010351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20250
I20260619 20:58:55.010449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:58:55.010650 392 external_mini_cluster.cc:1428] Setting key 01c1990362665780257650901b7b478f
I20260619 20:58:55.011499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:58:55.090631 20381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:55.090833 20381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:55.090853 20381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:55.091954 20381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:55.091993 20381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:55.092026 20381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:58:55.093333 20381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:55.093560 20381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:55.093850 20381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:55.098696 20387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:55.098696 20389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:55.098738 20386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:55.098912 20381 server_base.cc:1034] running on GCE node
I20260619 20:58:55.099138 20381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:55.099372 20381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:55.100531 20381 hybrid_clock.cc:648] HybridClock initialized: now 1781902735100524 us; error 44 us; skew 500 ppm
I20260619 20:58:55.101758 20381 webserver.cc:460] Webserver started at http://127.0.98.3:44413/ using document root <none> and password file <none>
I20260619 20:58:55.101982 20381 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:55.102056 20381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:55.102260 20381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:55.103276 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data/instance:
uuid: "464f310eaf9e4b8bab5c0fea8ac8003e"
format_stamp: "Formatted at 2026-06-19 20:58:55 on dist-test-slave-8bgz"
server_key: "f3bf4f7e1d0c71339beb709e52f20af6"
server_key_iv: "207639bb97e0832215acf071fed4c05e"
server_key_version: "encryptionkey@0"
I20260619 20:58:55.103618 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "464f310eaf9e4b8bab5c0fea8ac8003e"
format_stamp: "Formatted at 2026-06-19 20:58:55 on dist-test-slave-8bgz"
server_key: "f3bf4f7e1d0c71339beb709e52f20af6"
server_key_iv: "207639bb97e0832215acf071fed4c05e"
server_key_version: "encryptionkey@0"
I20260619 20:58:55.104861 20381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:58:55.105623 20395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:55.105851 20381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:58:55.105947 20381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal
uuid: "464f310eaf9e4b8bab5c0fea8ac8003e"
format_stamp: "Formatted at 2026-06-19 20:58:55 on dist-test-slave-8bgz"
server_key: "f3bf4f7e1d0c71339beb709e52f20af6"
server_key_iv: "207639bb97e0832215acf071fed4c05e"
server_key_version: "encryptionkey@0"
I20260619 20:58:55.106041 20381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:55.115860 20381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:55.116132 20381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:55.116276 20381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:55.116478 20381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:58:55.116791 20381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:58:55.116843 20381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:55.116881 20381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:58:55.116909 20381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:55.122711 20381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34013
I20260619 20:58:55.122761 20508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34013 every 8 connection(s)
I20260619 20:58:55.123063 20381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:58:55.126101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20381
I20260619 20:58:55.126205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:58:55.126412 392 external_mini_cluster.cc:1428] Setting key d995655437265b19b1c15ab478d820dc
I20260619 20:58:55.128434 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.123772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54069 (local address 127.0.98.62:37861)
0619 20:58:55.123814 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.123816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.124061 (+ 245us) server_negotiation.cc:407] Connection header received
0619 20:58:55.124607 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.124609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.124622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.124680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:55.125338 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.125946 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.127101 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.127399 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.127464 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.128216 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:55.128217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:55.128218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:55.128221 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:55.128233 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:55.128237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:55.128279 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.128360 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.128380 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:58:55.128646 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.123752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37861 (local address 127.0.98.3:54069)
0619 20:58:55.123935 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.123966 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.124482 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.124743 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.124747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.124757 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:58:55.125238 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.125242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.126040 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.126041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.127022 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.127026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.127478 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.127479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.127591 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.128079 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:58:55.128079 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:58:55.128117 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:58:55.128138 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:58:55.128304 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:58:55.128308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:58:55.128310 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.128354 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.128424 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260619 20:58:55.129076 20509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37861
I20260619 20:58:55.129179 20509 heartbeater.cc:464] Registering TS with master...
I20260619 20:58:55.129346 20509 heartbeater.cc:511] Master 127.0.98.62:37861 requested a full tablet report, sending...
I20260619 20:58:55.129721 20059 ts_manager.cc:194] Registered new tserver with Master: 464f310eaf9e4b8bab5c0fea8ac8003e (127.0.98.3:34013)
I20260619 20:58:55.130028 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54069
I20260619 20:58:55.138296 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:58:55.142359 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.139669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.98.62:37861)
0619 20:58:55.139720 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.139722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.139787 (+ 65us) server_negotiation.cc:407] Connection header received
0619 20:58:55.139833 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.139833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.139841 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.139886 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:55.140062 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.140550 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.141284 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.141496 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.141557 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.142118 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:55.142119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:55.142119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:55.142122 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:55.142134 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:55.142138 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:55.142181 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.142274 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.142310 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:58:55.143759 20059 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:58:55.145229 20059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:58:55.151053 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.0.98.3:34013)
0619 20:58:55.147507 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.147511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.147726 (+ 215us) server_negotiation.cc:407] Connection header received
0619 20:58:55.147817 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.147819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.147849 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.147937 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:55.148090 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.148844 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.150129 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.150693 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.150754 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.150812 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.150960 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.150998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260619 20:58:55.151054 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34013 (local address 127.0.0.1:51386)
0619 20:58:55.147594 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.147650 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.147732 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.147940 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.147942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.147944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:55.148008 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.148010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.148934 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.148935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.150014 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.150016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.150776 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.150777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.150856 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.150869 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.150943 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.150985 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":80,"threads_started":1}
I20260619 20:58:55.151724 20443 tablet_service.cc:1467] Processing CreateTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f (DEFAULT_TABLE table=test-table [id=b7a1322b221a4f31bc66d4be0bc6522e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:55.152079 20443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a32ee4bb94e359b9e69b9a97b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:55.152623 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43593 (local address 127.0.0.1:39550)
0619 20:58:55.147645 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.147657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.147753 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.148115 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.148115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.148117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:55.148156 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.148158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.150291 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.150291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.151388 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.151391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.152279 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.152280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.152444 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.152448 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.152493 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.152544 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":85,"threads_started":1}
I20260619 20:58:55.152853 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.0.98.2:43593)
0619 20:58:55.147574 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.147577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.147713 (+ 136us) server_negotiation.cc:407] Connection header received
0619 20:58:55.147871 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.147874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.147891 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.147949 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:55.148241 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.149398 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.151484 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.152188 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.152256 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.152338 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.152733 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.152774 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260619 20:58:55.152892 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38117 (local address 127.0.0.1:42968)
0619 20:58:55.147884 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.147894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.147940 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.148180 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.148181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.148183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:55.148216 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.148217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.149486 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.149487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.151621 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.151622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.152598 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.152599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.152662 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.152665 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.152775 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.152815 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":67,"threads_started":1}
I20260619 20:58:55.153107 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.147591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.0.98.1:38117)
0619 20:58:55.147811 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.147814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.147992 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:58:55.148034 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.148035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.148049 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.148117 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:55.148310 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.149399 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.151698 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.152369 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.152438 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.152525 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.152985 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.153028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260619 20:58:55.153810 20181 tablet_service.cc:1467] Processing CreateTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f (DEFAULT_TABLE table=test-table [id=b7a1322b221a4f31bc66d4be0bc6522e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:55.154162 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a32ee4bb94e359b9e69b9a97b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:55.154747 20312 tablet_service.cc:1467] Processing CreateTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f (DEFAULT_TABLE table=test-table [id=b7a1322b221a4f31bc66d4be0bc6522e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:58:55.155049 20312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93a32ee4bb94e359b9e69b9a97b982f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:55.155539 20523 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Bootstrap starting.
I20260619 20:58:55.156215 20523 tablet_bootstrap.cc:654] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:55.156529 20523 log.cc:824] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:55.157299 20523 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: No bootstrap required, opened a new log
I20260619 20:58:55.157370 20523 ts_tablet_manager.cc:1397] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:55.157853 20526 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Bootstrap starting.
I20260619 20:58:55.158368 20525 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: Bootstrap starting.
I20260619 20:58:55.158591 20526 tablet_bootstrap.cc:654] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:55.158893 20526 log.cc:824] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:55.159233 20525 tablet_bootstrap.cc:654] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:55.159523 20525 log.cc:824] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:55.159662 20526 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: No bootstrap required, opened a new log
I20260619 20:58:55.159734 20526 ts_tablet_manager.cc:1397] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:58:55.160169 20525 tablet_bootstrap.cc:492] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: No bootstrap required, opened a new log
I20260619 20:58:55.160249 20525 ts_tablet_manager.cc:1397] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:58:55.160130 20523 raft_consensus.cc:348] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.160292 20523 raft_consensus.cc:374] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:55.160342 20523 raft_consensus.cc:729] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464f310eaf9e4b8bab5c0fea8ac8003e, State: Initialized, Role: FOLLOWER
I20260619 20:58:55.160478 20523 consensus_queue.cc:260] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.160744 20523 ts_tablet_manager.cc:1428] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 20:58:55.160884 20509 heartbeater.cc:503] Master 127.0.98.62:37861 was elected leader, sending a full tablet report...
I20260619 20:58:55.162061 20526 raft_consensus.cc:348] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.162178 20526 raft_consensus.cc:374] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:55.162256 20526 raft_consensus.cc:729] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c418dab3dfd84b048bdbb6866b3673df, State: Initialized, Role: FOLLOWER
I20260619 20:58:55.162372 20526 consensus_queue.cc:260] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.162633 20526 ts_tablet_manager.cc:1428] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:58:55.162695 20378 heartbeater.cc:503] Master 127.0.98.62:37861 was elected leader, sending a full tablet report...
I20260619 20:58:55.162757 20525 raft_consensus.cc:348] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.162886 20525 raft_consensus.cc:374] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:55.162923 20525 raft_consensus.cc:729] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5da22923a87406e803f80e1a8b54f00, State: Initialized, Role: FOLLOWER
I20260619 20:58:55.163026 20525 consensus_queue.cc:260] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.163264 20525 ts_tablet_manager.cc:1428] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:58:55.163300 20247 heartbeater.cc:503] Master 127.0.98.62:37861 was elected leader, sending a full tablet report...
I20260619 20:58:55.221869 20530 raft_consensus.cc:482] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:58:55.221977 20530 raft_consensus.cc:504] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.222396 20530 leader_election.cc:288] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 464f310eaf9e4b8bab5c0fea8ac8003e (127.0.98.3:34013), f5da22923a87406e803f80e1a8b54f00 (127.0.98.1:38117)
I20260619 20:58:55.225101 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.222579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34013 (local address 127.0.98.2:43085)
0619 20:58:55.222649 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.222662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.222718 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.222880 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.222881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.222883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:55.222957 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.222959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.223583 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.223585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.224314 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.224316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.224849 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.224850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.224924 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.224927 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.224991 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.225032 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 20:58:55.225106 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.222655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43085 (local address 127.0.98.3:34013)
0619 20:58:55.222724 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.222727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.222759 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:58:55.222797 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.222799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.222809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.222855 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:55.223026 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.223495 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.224404 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.224766 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.224849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.224858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.224997 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.225046 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:58:55.225291 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.222877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34653 (local address 127.0.98.1:38117)
0619 20:58:55.222921 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.222923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.222939 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:58:55.222976 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.222976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.222985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.223048 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:58:55.223216 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.223672 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.224573 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.224931 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.225032 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.225042 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.225189 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.225227 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:58:55.225308 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.222707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38117 (local address 127.0.98.2:34653)
0619 20:58:55.222866 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.222876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:58:55.222916 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:58:55.223053 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:58:55.223055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:58:55.223057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:58:55.223134 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.223135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.223751 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.223752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.224480 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:58:55.224482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.225042 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.225043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:58:55.225109 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.225112 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:58:55.225184 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:58:55.225222 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260619 20:58:55.225447 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a32ee4bb94e359b9e69b9a97b982f" candidate_uuid: "c418dab3dfd84b048bdbb6866b3673df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" is_pre_election: true
I20260619 20:58:55.225605 20463 raft_consensus.cc:2454] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c418dab3dfd84b048bdbb6866b3673df in term 0.
I20260619 20:58:55.225827 20265 leader_election.cc:302] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464f310eaf9e4b8bab5c0fea8ac8003e, c418dab3dfd84b048bdbb6866b3673df; no voters:
I20260619 20:58:55.225965 20530 raft_consensus.cc:2781] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:58:55.226014 20530 raft_consensus.cc:482] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:58:55.225946 20201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a32ee4bb94e359b9e69b9a97b982f" candidate_uuid: "c418dab3dfd84b048bdbb6866b3673df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5da22923a87406e803f80e1a8b54f00" is_pre_election: true
I20260619 20:58:55.226043 20530 raft_consensus.cc:3037] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:55.226094 20201 raft_consensus.cc:2454] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c418dab3dfd84b048bdbb6866b3673df in term 0.
I20260619 20:58:55.226768 20530 raft_consensus.cc:504] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.226866 20530 leader_election.cc:288] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [CANDIDATE]: Term 1 election: Requested vote from peers 464f310eaf9e4b8bab5c0fea8ac8003e (127.0.98.3:34013), f5da22923a87406e803f80e1a8b54f00 (127.0.98.1:38117)
I20260619 20:58:55.227072 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a32ee4bb94e359b9e69b9a97b982f" candidate_uuid: "c418dab3dfd84b048bdbb6866b3673df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e"
I20260619 20:58:55.227082 20201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93a32ee4bb94e359b9e69b9a97b982f" candidate_uuid: "c418dab3dfd84b048bdbb6866b3673df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5da22923a87406e803f80e1a8b54f00"
I20260619 20:58:55.227169 20463 raft_consensus.cc:3037] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:55.227169 20201 raft_consensus.cc:3037] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:55.228037 20463 raft_consensus.cc:2454] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c418dab3dfd84b048bdbb6866b3673df in term 1.
I20260619 20:58:55.228046 20201 raft_consensus.cc:2454] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c418dab3dfd84b048bdbb6866b3673df in term 1.
I20260619 20:58:55.228230 20265 leader_election.cc:302] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c418dab3dfd84b048bdbb6866b3673df, f5da22923a87406e803f80e1a8b54f00; no voters:
I20260619 20:58:55.228364 20530 raft_consensus.cc:2781] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:55.228569 20530 raft_consensus.cc:686] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 1 LEADER]: Becoming Leader. State: Replica: c418dab3dfd84b048bdbb6866b3673df, State: Running, Role: LEADER
I20260619 20:58:55.228679 20530 consensus_queue.cc:237] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } }
I20260619 20:58:55.229512 20058 catalog_manager.cc:5495] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df reported cstate change: term changed from 0 to 1, leader changed from <none> to c418dab3dfd84b048bdbb6866b3673df (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c418dab3dfd84b048bdbb6866b3673df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c418dab3dfd84b048bdbb6866b3673df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 } health_report { overall_health: UNKNOWN } } }
I20260619 20:58:55.242705 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.240274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.98.2:43593)
0619 20:58:55.240314 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:55.240316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:55.240324 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:58:55.240409 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:55.240410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:55.240433 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:55.240508 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:58:55.240671 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.241123 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.241888 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:55.242093 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:55.242170 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:55.242369 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:58:55.242458 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:58:55.242547 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:55.242587 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 20:58:55.242606 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:58:55.244288 20201 raft_consensus.cc:1261] T e93a32ee4bb94e359b9e69b9a97b982f P f5da22923a87406e803f80e1a8b54f00 [term 1 FOLLOWER]: Refusing update from remote peer c418dab3dfd84b048bdbb6866b3673df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:55.244297 20463 raft_consensus.cc:1261] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 1 FOLLOWER]: Refusing update from remote peer c418dab3dfd84b048bdbb6866b3673df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:58:55.244606 20530 consensus_queue.cc:1035] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5da22923a87406e803f80e1a8b54f00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:55.244735 20533 consensus_queue.cc:1035] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [LEADER]: Connected to new peer: Peer: permanent_uuid: "464f310eaf9e4b8bab5c0fea8ac8003e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:58:55.247473 20537 mvcc.cc:205] Tried to move back new op lower bound from 7298673603558301696 to 7298673603497447424. Current Snapshot: MvccSnapshot[applied={T|T < 7298673603558301696}]
I20260619 20:58:55.249716 20058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:55.249807 20058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:58:55.250532 20058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 654b01629909499e88192fbb835da6d2: Sending DeleteTablet for 3 replicas of tablet e93a32ee4bb94e359b9e69b9a97b982f
I20260619 20:58:55.250835 20312 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:55 UTC) from {username='slave'} at 127.0.0.1:39550
I20260619 20:58:55.250887 20443 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:55 UTC) from {username='slave'} at 127.0.0.1:51386
I20260619 20:58:55.250887 20181 tablet_service.cc:1514] Processing DeleteTablet for tablet e93a32ee4bb94e359b9e69b9a97b982f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:58:55 UTC) from {username='slave'} at 127.0.0.1:42968
I20260619 20:58:55.251091 20547 tablet_replica.cc:330] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: stopping tablet replica
I20260619 20:58:55.251127 20546 tablet_replica.cc:330] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: stopping tablet replica
I20260619 20:58:55.251191 20547 raft_consensus.cc:2229] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:58:55.251214 20546 raft_consensus.cc:2229] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:58:55.251278 20547 raft_consensus.cc:2258] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:55.251317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20119
I20260619 20:58:55.251329 20546 raft_consensus.cc:2258] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:58:55.252246 20547 ts_tablet_manager.cc:1905] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:55.253702 20546 ts_tablet_manager.cc:1905] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:58:55.253863 20547 ts_tablet_manager.cc:1918] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:55.253943 20547 log.cc:1196] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-2/wal/wals/e93a32ee4bb94e359b9e69b9a97b982f
I20260619 20:58:55.254192 20547 ts_tablet_manager.cc:1939] T e93a32ee4bb94e359b9e69b9a97b982f P 464f310eaf9e4b8bab5c0fea8ac8003e: Deleting consensus metadata
I20260619 20:58:55.254598 20043 catalog_manager.cc:4841] TS 464f310eaf9e4b8bab5c0fea8ac8003e (127.0.98.3:34013): tablet e93a32ee4bb94e359b9e69b9a97b982f (table test-table [id=b7a1322b221a4f31bc66d4be0bc6522e]) successfully deleted
I20260619 20:58:55.255594 20546 ts_tablet_manager.cc:1918] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:58:55.255666 20546 log.cc:1196] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Deleting WAL directory at /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902497118594-392-0/minicluster-data/ts-1/wal/wals/e93a32ee4bb94e359b9e69b9a97b982f
I20260619 20:58:55.255903 20546 ts_tablet_manager.cc:1939] T e93a32ee4bb94e359b9e69b9a97b982f P c418dab3dfd84b048bdbb6866b3673df: Deleting consensus metadata
I20260619 20:58:55.256283 20046 catalog_manager.cc:4841] TS c418dab3dfd84b048bdbb6866b3673df (127.0.98.2:43593): tablet e93a32ee4bb94e359b9e69b9a97b982f (table test-table [id=b7a1322b221a4f31bc66d4be0bc6522e]) successfully deleted
W20260619 20:58:55.257427 20043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38117 (error 108)
I20260619 20:58:55.257655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20250
I20260619 20:58:55.257899 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:55.257759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38117 (local address 127.0.0.1:42978)
0619 20:58:55.257805 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:58:55.257844 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:58:55.258375 20043 catalog_manager.cc:4584] TS f5da22923a87406e803f80e1a8b54f00 (127.0.98.1:38117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e93a32ee4bb94e359b9e69b9a97b982f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38117: connect: Connection refused (error 111)
I20260619 20:58:55.262696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20381
I20260619 20:58:55.266834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20028
2026-06-19T20:58:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (694 ms)
[----------] 6 tests from EncryptionPolicyTest (8324 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:55 dist-test-slave-8bgz krb5kdc[20552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:55 dist-test-slave-8bgz krb5kdc[20552](info): set up 2 sockets
Jun 19 20:58:55 dist-test-slave-8bgz krb5kdc[20552](info): commencing operation
krb5kdc: starting...
W20260619 20:58:56.302186 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:58:56.420600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:56.501327 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:56.501528 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:56.501549 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:56.502727 20565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:56.502768 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:56.502784 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:56.502795 20565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:56.502807 20565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:56.504235 20565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:56.504434 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:56.504647 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:56.509310 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:56.509449 20565 server_base.cc:1034] running on GCE node
W20260619 20:58:56.509354 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:56.509641 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:56.509920 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:56.510171 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:56.511373 20565 hybrid_clock.cc:648] HybridClock initialized: now 1781902736511363 us; error 30 us; skew 500 ppm
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:56.513710 20565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:58:56.514273 20565 webserver.cc:460] Webserver started at http://127.0.0.1:42831/ using document root <none> and password file <none>
I20260619 20:58:56.514587 20565 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:56.514654 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:56.514783 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:56.515805 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "caf2e7d95af94f69a53e7a902a11d1f2"
format_stamp: "Formatted at 2026-06-19 20:58:56 on dist-test-slave-8bgz"
server_key: "673f484c2ad1e8e4f87c15a61989f278"
server_key_iv: "c37985b4237038401f2626fb47b94002"
server_key_version: "encryptionkey@0"
I20260619 20:58:56.516163 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "caf2e7d95af94f69a53e7a902a11d1f2"
format_stamp: "Formatted at 2026-06-19 20:58:56 on dist-test-slave-8bgz"
server_key: "673f484c2ad1e8e4f87c15a61989f278"
server_key_iv: "c37985b4237038401f2626fb47b94002"
server_key_version: "encryptionkey@0"
I20260619 20:58:56.517433 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:56.518139 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:56.518318 20565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:56.518416 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "caf2e7d95af94f69a53e7a902a11d1f2"
format_stamp: "Formatted at 2026-06-19 20:58:56 on dist-test-slave-8bgz"
server_key: "673f484c2ad1e8e4f87c15a61989f278"
server_key_iv: "c37985b4237038401f2626fb47b94002"
server_key_version: "encryptionkey@0"
I20260619 20:58:56.518497 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:56.533667 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:56.535553 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:56.535713 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:56.539793 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44307
I20260619 20:58:56.539829 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44307 every 8 connection(s)
I20260619 20:58:56.540180 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:56.540983 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:56.543191 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2: Bootstrap starting.
I20260619 20:58:56.543751 20633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:56.544042 20633 log.cc:824] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:56.544716 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2: No bootstrap required, opened a new log
I20260619 20:58:56.545548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20565
I20260619 20:58:56.545652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:56.545904 392 external_mini_cluster.cc:1428] Setting key 4d15626600fbc2ced2563f8c33a3d852
I20260619 20:58:56.546792 20633 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:58:56.546958 20633 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:56.547001 20633 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf2e7d95af94f69a53e7a902a11d1f2, State: Initialized, Role: FOLLOWER
I20260619 20:58:56.547137 20633 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:58:56.547220 20633 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:56.547269 20633 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:56.547317 20633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:56.548352 20633 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:58:56.548455 20633 leader_election.cc:302] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf2e7d95af94f69a53e7a902a11d1f2; no voters:
I20260619 20:58:56.548605 20633 leader_election.cc:288] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:56.548686 20638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:56.548820 20633 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:56.548840 20638 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [term 1 LEADER]: Becoming Leader. State: Replica: caf2e7d95af94f69a53e7a902a11d1f2, State: Running, Role: LEADER
I20260619 20:58:56.548920 20638 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:58:56.549343 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } } }
I20260619 20:58:56.549427 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:56.549670 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf2e7d95af94f69a53e7a902a11d1f2. Latest consensus state: current_term: 1 leader_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7d95af94f69a53e7a902a11d1f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } } }
I20260619 20:58:56.549680 20647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:56.549811 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:56.550386 20647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:56.552331 20647 catalog_manager.cc:1347] Generated new cluster ID: 54cc87b563034b2982ad1a5fd8cab425
I20260619 20:58:56.552385 20647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:58:56.556937 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:56.546488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36535 (local address 127.0.0.1:44307)
0619 20:58:56.546691 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:56.546701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:56.546735 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:58:56.546842 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:56.546849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:56.546900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:56.547054 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:56.547870 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:56.549300 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:56.550496 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:56.551033 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:56.551155 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:56.553655 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:56.553663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:56.553664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:56.553673 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:56.555419 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:56.555807 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:56.555810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:56.555810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:56.555917 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:56.556162 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:56.556163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:56.556164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:56.556416 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:56.556602 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:56.556689 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:58:56.556787 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260619 20:58:56.557827 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:58:56.565279 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:56.559215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54561 (local address 127.0.0.1:44307)
0619 20:58:56.559250 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:56.559252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:56.559373 (+ 121us) server_negotiation.cc:407] Connection header received
0619 20:58:56.559433 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:56.559433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:56.559443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:56.559492 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:56.560002 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:56.560478 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:56.561175 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:56.561415 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:56.561507 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:56.562703 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:56.562704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:56.562705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:58:56.562711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:56.564111 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:56.564475 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:56.564476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:56.564477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:56.564528 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:58:56.564750 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:58:56.564751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:58:56.564752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:58:56.564859 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:56.564978 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:56.565105 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:58:56.565166 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:58:56.567410 20647 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:56.567963 20647 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:56.574097 20647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf2e7d95af94f69a53e7a902a11d1f2: Generated new TSK 0
I20260619 20:58:56.574296 20647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:56.577976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20565
W20260619 20:58:56.582132 20550 connection.cc:535] client connection to 127.0.0.1:44307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44307): Connection reset by peer (error 104)
2026-06-19T20:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20664](info): set up 2 sockets
Jun 19 20:58:56 dist-test-slave-8bgz krb5kdc[20664](info): commencing operation
krb5kdc: starting...
W20260619 20:58:57.606070 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:57 dist-test-slave-8bgz krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:58:57.715495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:57.794746 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:57.794967 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:57.795006 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:57.796149 20677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:57.796213 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:57.796241 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:57.796273 20677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:57.796303 20677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:57.797720 20677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:57.797984 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:57.798251 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:57.802983 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:57.803005 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:57.803079 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:57.803269 20677 server_base.cc:1034] running on GCE node
I20260619 20:58:57.803483 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:57.803846 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:57.805003 20677 hybrid_clock.cc:648] HybridClock initialized: now 1781902737804982 us; error 35 us; skew 500 ppm
Jun 19 20:58:57 dist-test-slave-8bgz krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:57.811882 20677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:58:57.812342 20677 webserver.cc:460] Webserver started at http://127.0.0.1:42073/ using document root <none> and password file <none>
I20260619 20:58:57.812595 20677 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:57.812654 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:57.812741 20677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:57.813691 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "32efed31f69a442c9faa9d7e9fa8abc0"
format_stamp: "Formatted at 2026-06-19 20:58:57 on dist-test-slave-8bgz"
server_key: "88d9b3d7bb44266464e888e733731727"
server_key_iv: "a863e3079dbb2822e621c9d1a43a5daf"
server_key_version: "encryptionkey@0"
I20260619 20:58:57.813987 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "32efed31f69a442c9faa9d7e9fa8abc0"
format_stamp: "Formatted at 2026-06-19 20:58:57 on dist-test-slave-8bgz"
server_key: "88d9b3d7bb44266464e888e733731727"
server_key_iv: "a863e3079dbb2822e621c9d1a43a5daf"
server_key_version: "encryptionkey@0"
I20260619 20:58:57.815265 20677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:57.815901 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:57.816059 20677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:58:57.816126 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "32efed31f69a442c9faa9d7e9fa8abc0"
format_stamp: "Formatted at 2026-06-19 20:58:57 on dist-test-slave-8bgz"
server_key: "88d9b3d7bb44266464e888e733731727"
server_key_iv: "a863e3079dbb2822e621c9d1a43a5daf"
server_key_version: "encryptionkey@0"
I20260619 20:58:57.816179 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:57.847548 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:57.849565 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:57.849673 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:57.853592 20677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41631
I20260619 20:58:57.853673 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41631 every 8 connection(s)
I20260619 20:58:57.853984 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:57.854876 20745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:57.857070 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0: Bootstrap starting.
I20260619 20:58:57.857620 20745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:57.857861 20745 log.cc:824] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:57.858616 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0: No bootstrap required, opened a new log
I20260619 20:58:57.859555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20677
I20260619 20:58:57.859678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:57.859851 392 external_mini_cluster.cc:1428] Setting key a2f399fd916e0c4e4ec2a2cd19593d0d
I20260619 20:58:57.860504 20745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } }
I20260619 20:58:57.860675 20745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:57.860708 20745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32efed31f69a442c9faa9d7e9fa8abc0, State: Initialized, Role: FOLLOWER
I20260619 20:58:57.860829 20745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } }
I20260619 20:58:57.860913 20745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:57.860944 20745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:57.860987 20745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:57.861704 20745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } }
I20260619 20:58:57.861809 20745 leader_election.cc:302] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32efed31f69a442c9faa9d7e9fa8abc0; no voters:
I20260619 20:58:57.861967 20745 leader_election.cc:288] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:57.862030 20750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:57.862159 20750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [term 1 LEADER]: Becoming Leader. State: Replica: 32efed31f69a442c9faa9d7e9fa8abc0, State: Running, Role: LEADER
I20260619 20:58:57.862185 20745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:57.862267 20750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } }
I20260619 20:58:57.862584 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32efed31f69a442c9faa9d7e9fa8abc0. Latest consensus state: current_term: 1 leader_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } } }
I20260619 20:58:57.862659 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:57.862602 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32efed31f69a442c9faa9d7e9fa8abc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41631 } } }
I20260619 20:58:57.862782 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:57.862908 20757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:57.863404 20757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:57.864804 20757 catalog_manager.cc:1347] Generated new cluster ID: 6805fc63a65149dca97052153c9ef1ae
I20260619 20:58:57.864856 20757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:57.865864 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:57.860338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48177 (local address 127.0.0.1:41631)
0619 20:58:57.860546 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:57.860555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:57.860585 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:58:57.860689 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:57.860696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:57.860752 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:57.860887 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:57.861119 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:57.862865 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:57.863909 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:57.864362 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:57.864478 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:57.865206 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:57.865210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:57.865210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:57.865408 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:57.865443 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:57.865457 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:57.865544 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:57.865650 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:58:57.865674 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260619 20:58:57.867164 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:58:57.872128 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:57.868672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57673 (local address 127.0.0.1:41631)
0619 20:58:57.868707 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:57.868709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:57.868848 (+ 139us) server_negotiation.cc:407] Connection header received
0619 20:58:57.868875 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:57.868876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:57.868900 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:57.868964 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:57.869099 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:57.869839 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:57.870815 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:58:57.871155 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:58:57.871258 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:58:57.871882 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:57.871884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:57.871884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:57.871889 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:57.871902 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:57.871907 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:57.871972 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:57.872042 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:58:57.872055 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:58:57.881177 20757 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:57.881695 20757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:57.889111 20757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32efed31f69a442c9faa9d7e9fa8abc0: Generated new TSK 0
I20260619 20:58:57.889294 20757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:57.899369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20677
W20260619 20:58:57.903419 20662 connection.cc:535] client connection to 127.0.0.1:41631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41631): Connection reset by peer (error 104)
2026-06-19T20:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:57 dist-test-slave-8bgz krb5kdc[20776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:57 dist-test-slave-8bgz krb5kdc[20776](info): set up 2 sockets
Jun 19 20:58:57 dist-test-slave-8bgz krb5kdc[20776](info): commencing operation
krb5kdc: starting...
W20260619 20:58:58.930085 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:58:59 dist-test-slave-8bgz krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:58:59.044131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:58:59.121835 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:58:59.122028 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:58:59.122051 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:58:59.123216 20789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:58:59.123258 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:58:59.123272 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:58:59.123311 20789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:58:59.123342 20789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:58:59.124724 20789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:58:59.124970 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:58:59.125188 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:58:59.129901 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:58:59.129920 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:59.129951 20789 server_base.cc:1034] running on GCE node
W20260619 20:58:59.129948 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:58:59.130306 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:58:59.130558 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:58:59.131723 20789 hybrid_clock.cc:648] HybridClock initialized: now 1781902739131707 us; error 32 us; skew 500 ppm
Jun 19 20:58:59 dist-test-slave-8bgz krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:58:59.134055 20789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:58:59.134536 20789 webserver.cc:460] Webserver started at http://127.0.0.1:45663/ using document root <none> and password file <none>
I20260619 20:58:59.134766 20789 fs_manager.cc:359] Metadata directory not provided
I20260619 20:58:59.134831 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:58:59.134987 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:58:59.135980 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "420e89275e27426bbdc2ac455b31e7b9"
format_stamp: "Formatted at 2026-06-19 20:58:59 on dist-test-slave-8bgz"
server_key: "bf268495ad717214fd543fe974812bf5"
server_key_iv: "5dbfe2227106f0a14920b5534d2c53e1"
server_key_version: "encryptionkey@0"
I20260619 20:58:59.136303 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "420e89275e27426bbdc2ac455b31e7b9"
format_stamp: "Formatted at 2026-06-19 20:58:59 on dist-test-slave-8bgz"
server_key: "bf268495ad717214fd543fe974812bf5"
server_key_iv: "5dbfe2227106f0a14920b5534d2c53e1"
server_key_version: "encryptionkey@0"
I20260619 20:58:59.137499 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:58:59.138257 20804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:58:59.138433 20789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:58:59.138528 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "420e89275e27426bbdc2ac455b31e7b9"
format_stamp: "Formatted at 2026-06-19 20:58:59 on dist-test-slave-8bgz"
server_key: "bf268495ad717214fd543fe974812bf5"
server_key_iv: "5dbfe2227106f0a14920b5534d2c53e1"
server_key_version: "encryptionkey@0"
I20260619 20:58:59.138612 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:58:59.141736 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:58:59.143466 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:58:59.143615 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:58:59.147363 20789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46509
I20260619 20:58:59.147389 20856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46509 every 8 connection(s)
I20260619 20:58:59.147675 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:58:59.148270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20789
I20260619 20:58:59.148407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:58:59.148617 392 external_mini_cluster.cc:1428] Setting key 950caebf875b583ed77e15c35eab01df
I20260619 20:58:59.148762 20857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:58:59.150969 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:59.149101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37669 (local address 127.0.0.1:46509)
0619 20:58:59.149280 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:59.149288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:59.149320 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:58:59.149810 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:59.149816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:59.149839 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:59.149943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:59.150552 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:59.150555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:59.150556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:59.150559 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:59.150591 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:59.150600 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:59.150653 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:59.150738 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:58:59.150785 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260619 20:58:59.151942 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:58:59.152117 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9: Bootstrap starting.
I20260619 20:58:59.152650 20857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9: Neither blocks nor log segments found. Creating new log.
I20260619 20:58:59.152931 20857 log.cc:824] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9: Log is configured to *not* fsync() on all Append() calls
I20260619 20:58:59.153592 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9: No bootstrap required, opened a new log
I20260619 20:58:59.155298 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:58:59.153315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47995 (local address 127.0.0.1:46509)
0619 20:58:59.153351 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:58:59.153353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:58:59.154385 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 20:58:59.154449 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:58:59.154450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:58:59.154470 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:58:59.154524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:58:59.155084 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:58:59.155085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:58:59.155085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:58:59.155089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:58:59.155099 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:58:59.155103 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:58:59.155142 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:58:59.155224 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:58:59.155238 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:58:59.155615 20857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260619 20:58:59.155790 20857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:58:59.155840 20857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420e89275e27426bbdc2ac455b31e7b9, State: Initialized, Role: FOLLOWER
I20260619 20:58:59.155956 20857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260619 20:58:59.156061 20857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:58:59.156103 20857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:58:59.156149 20857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:58:59.156605 20857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260619 20:58:59.156714 20857 leader_election.cc:302] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 420e89275e27426bbdc2ac455b31e7b9; no voters:
I20260619 20:58:59.156879 20857 leader_election.cc:288] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:58:59.156908 20867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:58:59.157042 20867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [term 1 LEADER]: Becoming Leader. State: Replica: 420e89275e27426bbdc2ac455b31e7b9, State: Running, Role: LEADER
I20260619 20:58:59.157122 20867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260619 20:58:59.157135 20857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:58:59.157447 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 420e89275e27426bbdc2ac455b31e7b9. Latest consensus state: current_term: 1 leader_uuid: "420e89275e27426bbdc2ac455b31e7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } } }
I20260619 20:58:59.157459 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "420e89275e27426bbdc2ac455b31e7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420e89275e27426bbdc2ac455b31e7b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } } }
I20260619 20:58:59.157567 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:59.157574 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9 [sys.catalog]: This master's current role is: LEADER
I20260619 20:58:59.157922 20875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:58:59.158388 20875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:58:59.159825 20875 catalog_manager.cc:1347] Generated new cluster ID: 071b917faf184d4fa21029cb56e8f5fa
I20260619 20:58:59.159878 20875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:58:59.180826 20875 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:58:59.180990 20875 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:58:59.189193 20875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 420e89275e27426bbdc2ac455b31e7b9: Generated new TSK 0
I20260619 20:58:59.189376 20875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:58:59.190871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20789
2026-06-19T20:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:58:59 dist-test-slave-8bgz krb5kdc[20888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:58:59 dist-test-slave-8bgz krb5kdc[20888](info): set up 2 sockets
Jun 19 20:58:59 dist-test-slave-8bgz krb5kdc[20888](info): commencing operation
krb5kdc: starting...
W20260619 20:59:00.222309 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:00.334055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:00.415802 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:00.415984 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:00.416005 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:00.417141 20901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:00.417176 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:00.417191 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:00.417203 20901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:00.417215 20901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:00.418576 20901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:00.418742 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:00.418892 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:00.423381 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:00.423377 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:00.423444 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:00.423552 20901 server_base.cc:1034] running on GCE node
I20260619 20:59:00.423841 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:00.424114 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:00.425258 20901 hybrid_clock.cc:648] HybridClock initialized: now 1781902740425240 us; error 34 us; skew 500 ppm
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:00.427666 20901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:00.428162 20901 webserver.cc:460] Webserver started at http://127.0.0.1:35159/ using document root <none> and password file <none>
I20260619 20:59:00.428398 20901 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:00.428469 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:00.428630 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:00.429654 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "9d8db46bc86045b596bd6bec6f7eb7ae"
format_stamp: "Formatted at 2026-06-19 20:59:00 on dist-test-slave-8bgz"
server_key: "6556c50a6220f27937b144263e7294ca"
server_key_iv: "f9b3ea30b9b3b1f63ffdc6a729faff7f"
server_key_version: "encryptionkey@0"
I20260619 20:59:00.430032 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d8db46bc86045b596bd6bec6f7eb7ae"
format_stamp: "Formatted at 2026-06-19 20:59:00 on dist-test-slave-8bgz"
server_key: "6556c50a6220f27937b144263e7294ca"
server_key_iv: "f9b3ea30b9b3b1f63ffdc6a729faff7f"
server_key_version: "encryptionkey@0"
I20260619 20:59:00.431942 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 20:59:00.432718 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:00.432896 20901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:59:00.433007 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "9d8db46bc86045b596bd6bec6f7eb7ae"
format_stamp: "Formatted at 2026-06-19 20:59:00 on dist-test-slave-8bgz"
server_key: "6556c50a6220f27937b144263e7294ca"
server_key_iv: "f9b3ea30b9b3b1f63ffdc6a729faff7f"
server_key_version: "encryptionkey@0"
I20260619 20:59:00.433095 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:00.446447 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:00.448225 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:00.448391 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:00.452301 20901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43383
I20260619 20:59:00.452332 20968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43383 every 8 connection(s)
I20260619 20:59:00.452670 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:00.453470 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:00.455693 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: Bootstrap starting.
I20260619 20:59:00.456243 20969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:00.456527 20969 log.cc:824] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:00.457234 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: No bootstrap required, opened a new log
I20260619 20:59:00.458030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 20901
I20260619 20:59:00.458173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:00.458382 392 external_mini_cluster.cc:1428] Setting key 4f7cef20480ad8531d9b6e0c1458bee0
I20260619 20:59:00.459189 20969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 20:59:00.459342 20969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:00.459401 20969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8db46bc86045b596bd6bec6f7eb7ae, State: Initialized, Role: FOLLOWER
I20260619 20:59:00.459545 20969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 20:59:00.459632 20969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:00.459671 20969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:00.459721 20969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:00.460518 20969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 20:59:00.460634 20969 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8db46bc86045b596bd6bec6f7eb7ae; no voters:
I20260619 20:59:00.460806 20969 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:00.460855 20974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:00.461005 20974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [term 1 LEADER]: Becoming Leader. State: Replica: 9d8db46bc86045b596bd6bec6f7eb7ae, State: Running, Role: LEADER
I20260619 20:59:00.461066 20969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:00.461151 20974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 20:59:00.462402 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260619 20:59:00.462406 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8db46bc86045b596bd6bec6f7eb7ae. Latest consensus state: current_term: 1 leader_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8db46bc86045b596bd6bec6f7eb7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260619 20:59:00.462733 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:00.462729 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae [sys.catalog]: This master's current role is: LEADER
W20260619 20:59:00.463202 20989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:00.463306 20989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:00.463387 20990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:00.463553 20990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:00.465037 20990 catalog_manager.cc:1347] Generated new cluster ID: 54c82ed3af0c4c5e90a4ece5c4c41f55
I20260619 20:59:00.465090 20990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[20888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:59:00.469367 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:00.458900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37673 (local address 127.0.0.1:43383)
0619 20:59:00.459071 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:00.459094 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:00.459119 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:59:00.459209 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:00.459214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:00.459253 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:00.459391 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:00.460119 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:00.461201 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:00.462417 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:00.463068 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:00.463180 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:00.465830 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:00.465833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:00.465834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:00.465846 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:00.467579 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:00.467950 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:00.467953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:00.467954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:00.468066 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:00.468270 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:00.468271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:00.468272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:00.468499 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:00.469085 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:00.469139 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 20:59:00.469236 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260619 20:59:00.470248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:00.480885 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:00.474474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33235 (local address 127.0.0.1:43383)
0619 20:59:00.474522 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:00.474524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:00.474642 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:59:00.474714 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:00.474715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:00.474725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:00.474779 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:00.475282 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:00.475757 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:00.476468 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:00.476704 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:00.476780 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:00.477896 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:00.477897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:00.477898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:00.477905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:00.479722 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:00.480116 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:00.480117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:00.480118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:00.480171 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:00.480396 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:00.480397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:00.480398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:00.480509 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:00.480625 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:00.480718 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:59:00.480789 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:59:00.488540 20990 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:00.489032 20990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:00.492204 20990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8db46bc86045b596bd6bec6f7eb7ae: Generated new TSK 0
I20260619 20:59:00.492372 20990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:00.496306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 20901
2026-06-19T20:59:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[21000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[21000](info): set up 2 sockets
Jun 19 20:59:00 dist-test-slave-8bgz krb5kdc[21000](info): commencing operation
krb5kdc: starting...
W20260619 20:59:01.530153 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:01 dist-test-slave-8bgz krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:01.640789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:01.719264 21013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:01.719485 21013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:01.719521 21013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:01.720642 21013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:01.720706 21013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:01.720731 21013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:01.720759 21013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:01.720784 21013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:01.722249 21013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:01.722507 21013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:01.722728 21013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:01.727452 21018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:01.727497 21021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:01.727535 21019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:01.727561 21013 server_base.cc:1034] running on GCE node
I20260619 20:59:01.727931 21013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:01.728255 21013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:01.729417 21013 hybrid_clock.cc:648] HybridClock initialized: now 1781902741729406 us; error 41 us; skew 500 ppm
Jun 19 20:59:01 dist-test-slave-8bgz krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:01.731801 21013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:01.732203 21013 webserver.cc:460] Webserver started at http://127.0.0.1:35995/ using document root <none> and password file <none>
I20260619 20:59:01.732425 21013 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:01.732494 21013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:01.732641 21013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:01.733642 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e"
format_stamp: "Formatted at 2026-06-19 20:59:01 on dist-test-slave-8bgz"
server_key: "2d2e24165290b191f5d590d49bcdc0da"
server_key_iv: "ee035894d5bab52bd32f71117ede08b0"
server_key_version: "encryptionkey@0"
I20260619 20:59:01.733994 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e"
format_stamp: "Formatted at 2026-06-19 20:59:01 on dist-test-slave-8bgz"
server_key: "2d2e24165290b191f5d590d49bcdc0da"
server_key_iv: "ee035894d5bab52bd32f71117ede08b0"
server_key_version: "encryptionkey@0"
I20260619 20:59:01.735328 21013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:59:01.736014 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:01.736173 21013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:01.736266 21013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e"
format_stamp: "Formatted at 2026-06-19 20:59:01 on dist-test-slave-8bgz"
server_key: "2d2e24165290b191f5d590d49bcdc0da"
server_key_iv: "ee035894d5bab52bd32f71117ede08b0"
server_key_version: "encryptionkey@0"
I20260619 20:59:01.736351 21013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:01.749053 21013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:01.750808 21013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:01.750964 21013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:01.754679 21013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42411
I20260619 20:59:01.754717 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42411 every 8 connection(s)
I20260619 20:59:01.755226 21013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:01.755865 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:01.758080 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e: Bootstrap starting.
I20260619 20:59:01.758661 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:01.758939 21081 log.cc:824] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:01.759608 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e: No bootstrap required, opened a new log
I20260619 20:59:01.761116 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260619 20:59:01.761245 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:01.761266 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6f3d8f90d64af5b8b3b8dc98fa727e, State: Initialized, Role: FOLLOWER
I20260619 20:59:01.761416 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260619 20:59:01.761477 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:01.761498 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:01.761555 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:01.762509 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260619 20:59:01.762621 21081 leader_election.cc:302] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af6f3d8f90d64af5b8b3b8dc98fa727e; no voters:
I20260619 20:59:01.762791 21081 leader_election.cc:288] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:01.762835 21084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:01.763002 21084 raft_consensus.cc:686] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [term 1 LEADER]: Becoming Leader. State: Replica: af6f3d8f90d64af5b8b3b8dc98fa727e, State: Running, Role: LEADER
I20260619 20:59:01.763064 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:01.763113 21084 consensus_queue.cc:237] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } }
I20260619 20:59:01.763514 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [sys.catalog]: SysCatalogTable state changed. Reason: New leader af6f3d8f90d64af5b8b3b8dc98fa727e. Latest consensus state: current_term: 1 leader_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260619 20:59:01.763598 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:01.763499 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6f3d8f90d64af5b8b3b8dc98fa727e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42411 } } }
I20260619 20:59:01.763777 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:01.763862 21093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:01.764536 21093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:01.764767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21013
I20260619 20:59:01.764914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:01.765087 392 external_mini_cluster.cc:1428] Setting key 07040e3c78ba9bbbdfffbafeb1e7eaf0
I20260619 20:59:01.766189 21093 catalog_manager.cc:1347] Generated new cluster ID: 462f23e11bd74a7aa52a384978e73f82
I20260619 20:59:01.766259 21093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:01.771132 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:01.765502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38647 (local address 127.0.0.1:42411)
0619 20:59:01.765894 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:01.765903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:01.765934 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:59:01.766041 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:01.766047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:01.766101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:01.766268 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:01.766467 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:01.767812 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:01.768927 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:01.769532 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:01.769629 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:01.770338 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:01.770342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:01.770342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:01.770553 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:01.770593 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:01.770607 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:01.770673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:01.770758 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:59:01.770918 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":341,"thread_start_us":66,"threads_started":1}
I20260619 20:59:01.772013 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:01.776772 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:01.773352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38295 (local address 127.0.0.1:42411)
0619 20:59:01.773387 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:01.773389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:01.773585 (+ 196us) server_negotiation.cc:407] Connection header received
0619 20:59:01.773606 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:01.773607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:01.773617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:01.773667 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:01.773766 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:01.774486 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:01.775492 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:01.775802 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:01.775913 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:01.776528 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:01.776529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:01.776529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:01.776533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:01.776546 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:01.776550 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:01.776605 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:01.776680 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 20:59:01.776703 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:01.781042 21093 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:01.781561 21093 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:01.794982 21093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af6f3d8f90d64af5b8b3b8dc98fa727e: Generated new TSK 0
I20260619 20:59:01.795166 21093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:01.800365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21013
2026-06-19T20:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:01 dist-test-slave-8bgz krb5kdc[21112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:01 dist-test-slave-8bgz krb5kdc[21112](info): set up 2 sockets
Jun 19 20:59:01 dist-test-slave-8bgz krb5kdc[21112](info): commencing operation
krb5kdc: starting...
W20260619 20:59:02.834148 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:02 dist-test-slave-8bgz krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:02.946719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:03.024544 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:03.024741 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:03.024763 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:03.025906 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:03.025955 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:03.025969 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:03.025981 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:03.025993 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:03.027465 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:03.027714 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:03.027938 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:03.032706 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:03.032714 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:03.032835 21125 server_base.cc:1034] running on GCE node
W20260619 20:59:03.032714 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:03.033164 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:03.033486 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:03.034662 21125 hybrid_clock.cc:648] HybridClock initialized: now 1781902743034638 us; error 38 us; skew 500 ppm
Jun 19 20:59:03 dist-test-slave-8bgz krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:03.040076 21125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:03.040585 21125 webserver.cc:460] Webserver started at http://127.0.0.1:44297/ using document root <none> and password file <none>
I20260619 20:59:03.040823 21125 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:03.040891 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:03.041041 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:03.042070 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "a50f2a848ba94bb4864e624384eca4a8"
format_stamp: "Formatted at 2026-06-19 20:59:03 on dist-test-slave-8bgz"
server_key: "11c508fe947bd6e6c5eb008f6dc4a332"
server_key_iv: "5472c4df85967cb34885eff53574ca6e"
server_key_version: "encryptionkey@0"
I20260619 20:59:03.042464 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "a50f2a848ba94bb4864e624384eca4a8"
format_stamp: "Formatted at 2026-06-19 20:59:03 on dist-test-slave-8bgz"
server_key: "11c508fe947bd6e6c5eb008f6dc4a332"
server_key_iv: "5472c4df85967cb34885eff53574ca6e"
server_key_version: "encryptionkey@0"
I20260619 20:59:03.043744 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:03.044531 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:03.044710 21125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:03.044811 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "a50f2a848ba94bb4864e624384eca4a8"
format_stamp: "Formatted at 2026-06-19 20:59:03 on dist-test-slave-8bgz"
server_key: "11c508fe947bd6e6c5eb008f6dc4a332"
server_key_iv: "5472c4df85967cb34885eff53574ca6e"
server_key_version: "encryptionkey@0"
I20260619 20:59:03.044893 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:03.048182 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:03.050477 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:03.050628 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:03.054519 21125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42523
I20260619 20:59:03.054550 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42523 every 8 connection(s)
I20260619 20:59:03.054854 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:03.055742 21193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:03.057812 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8: Bootstrap starting.
I20260619 20:59:03.058403 21193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:03.058629 21193 log.cc:824] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:03.059245 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8: No bootstrap required, opened a new log
I20260619 20:59:03.060729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21125
I20260619 20:59:03.060856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:03.061046 392 external_mini_cluster.cc:1428] Setting key 3bef22d4be51fcccefc12aa547ee8918
I20260619 20:59:03.061012 21193 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260619 20:59:03.061178 21193 raft_consensus.cc:374] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:03.061223 21193 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50f2a848ba94bb4864e624384eca4a8, State: Initialized, Role: FOLLOWER
I20260619 20:59:03.061390 21193 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260619 20:59:03.061477 21193 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:03.061535 21193 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:03.061627 21193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:03.062794 21193 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260619 20:59:03.062917 21193 leader_election.cc:302] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50f2a848ba94bb4864e624384eca4a8; no voters:
I20260619 20:59:03.063068 21193 leader_election.cc:288] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:03.063098 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:03.061505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47217 (local address 127.0.0.1:42523)
0619 20:59:03.061669 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:03.061678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:03.061711 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:59:03.061849 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:03.061854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:03.061876 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:03.061984 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:03.062621 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:03.062624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:03.062624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:03.062628 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:03.062662 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:03.062672 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:03.062726 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:03.062831 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 20:59:03.062889 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260619 20:59:03.063119 21198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:03.063309 21193 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:03.063373 21198 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [term 1 LEADER]: Becoming Leader. State: Replica: a50f2a848ba94bb4864e624384eca4a8, State: Running, Role: LEADER
I20260619 20:59:03.063482 21198 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } }
I20260619 20:59:03.063851 21198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50f2a848ba94bb4864e624384eca4a8. Latest consensus state: current_term: 1 leader_uuid: "a50f2a848ba94bb4864e624384eca4a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } } }
I20260619 20:59:03.063896 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a50f2a848ba94bb4864e624384eca4a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50f2a848ba94bb4864e624384eca4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42523 } } }
I20260619 20:59:03.064011 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:03.064029 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:03.064208 21198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:03.064599 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:03.065213 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:03.066711 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:03.065510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52759 (local address 127.0.0.1:42523)
0619 20:59:03.065542 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:03.065543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:03.065668 (+ 125us) server_negotiation.cc:407] Connection header received
0619 20:59:03.065697 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:03.065698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:03.065702 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:03.065764 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:03.066477 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:03.066478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:03.066478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:03.066482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:03.066495 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:03.066499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:03.066559 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:03.066643 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 20:59:03.066657 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:59:03.066910 21207 catalog_manager.cc:1347] Generated new cluster ID: 041911a779a64282b5315a91a3b9c07e
I20260619 20:59:03.066972 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:03.084537 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:03.084743 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:03.088243 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a50f2a848ba94bb4864e624384eca4a8: Generated new TSK 0
I20260619 20:59:03.088409 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:03.096491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21125
2026-06-19T20:59:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:03 dist-test-slave-8bgz krb5kdc[21224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:03 dist-test-slave-8bgz krb5kdc[21224](info): set up 2 sockets
Jun 19 20:59:03 dist-test-slave-8bgz krb5kdc[21224](info): commencing operation
krb5kdc: starting...
W20260619 20:59:04.118386 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:04.228950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:04.307245 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:04.307484 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:04.307525 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:04.308692 21237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:04.308764 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:04.308789 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:04.308816 21237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:04.308847 21237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:04.310308 21237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:04.310576 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:04.310793 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:04.315542 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:04.315620 21237 server_base.cc:1034] running on GCE node
W20260619 20:59:04.315532 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:04.315531 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:04.315992 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:04.316341 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:04.317507 21237 hybrid_clock.cc:648] HybridClock initialized: now 1781902744317493 us; error 28 us; skew 500 ppm
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:04.319970 21237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:04.320363 21237 webserver.cc:460] Webserver started at http://127.0.0.1:39909/ using document root <none> and password file <none>
I20260619 20:59:04.320576 21237 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:04.320642 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:04.320787 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:04.321775 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "04a23a6d959c435eb9e5301102cb1719"
format_stamp: "Formatted at 2026-06-19 20:59:04 on dist-test-slave-8bgz"
server_key: "70715c1e742cf916d8c94954ba23336e"
server_key_iv: "b92046dee7dd6502ae6ba80080f5df40"
server_key_version: "encryptionkey@0"
I20260619 20:59:04.322130 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "04a23a6d959c435eb9e5301102cb1719"
format_stamp: "Formatted at 2026-06-19 20:59:04 on dist-test-slave-8bgz"
server_key: "70715c1e742cf916d8c94954ba23336e"
server_key_iv: "b92046dee7dd6502ae6ba80080f5df40"
server_key_version: "encryptionkey@0"
I20260619 20:59:04.323413 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:59:04.324154 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:04.324333 21237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:59:04.324434 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "04a23a6d959c435eb9e5301102cb1719"
format_stamp: "Formatted at 2026-06-19 20:59:04 on dist-test-slave-8bgz"
server_key: "70715c1e742cf916d8c94954ba23336e"
server_key_iv: "b92046dee7dd6502ae6ba80080f5df40"
server_key_version: "encryptionkey@0"
I20260619 20:59:04.324515 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:04.337771 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:04.340018 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:04.340178 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:04.344203 21237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32819
I20260619 20:59:04.344236 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32819 every 8 connection(s)
I20260619 20:59:04.344561 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:04.345352 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:04.347774 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: Bootstrap starting.
I20260619 20:59:04.348333 21305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:04.348620 21305 log.cc:824] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:04.349381 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: No bootstrap required, opened a new log
I20260619 20:59:04.351089 21305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } }
I20260619 20:59:04.351220 21305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:04.351241 21305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a23a6d959c435eb9e5301102cb1719, State: Initialized, Role: FOLLOWER
I20260619 20:59:04.351434 21305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } }
I20260619 20:59:04.351516 21305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:04.351555 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:04.351603 21305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:04.352097 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } }
I20260619 20:59:04.352203 21305 leader_election.cc:302] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a23a6d959c435eb9e5301102cb1719; no voters:
I20260619 20:59:04.352381 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:04.352450 21308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:04.352638 21305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:04.352643 21308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [term 1 LEADER]: Becoming Leader. State: Replica: 04a23a6d959c435eb9e5301102cb1719, State: Running, Role: LEADER
I20260619 20:59:04.352829 21308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } }
I20260619 20:59:04.353243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21237
I20260619 20:59:04.353376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:04.353587 392 external_mini_cluster.cc:1428] Setting key 5a5b76345e06d33cf2e3637e90091944
I20260619 20:59:04.354135 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a23a6d959c435eb9e5301102cb1719. Latest consensus state: current_term: 1 leader_uuid: "04a23a6d959c435eb9e5301102cb1719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } } }
I20260619 20:59:04.354283 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:04.354641 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a23a6d959c435eb9e5301102cb1719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a23a6d959c435eb9e5301102cb1719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32819 } } }
I20260619 20:59:04.354748 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719 [sys.catalog]: This master's current role is: LEADER
W20260619 20:59:04.354776 21324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:04.354894 21324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:04.354987 21326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:04.355151 21326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:04.356640 21326 catalog_manager.cc:1347] Generated new cluster ID: e906dcd1bce740c38c92a0d6ef80c3c1
I20260619 20:59:04.356688 21326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:59:04.364914 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:04.354237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:46333 (local address 127.0.0.1:32819)
0619 20:59:04.354436 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:04.354445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:04.354477 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:59:04.354606 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:04.354613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:04.354670 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:04.354853 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:04.355589 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:04.356841 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:04.357874 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:04.358462 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:04.358553 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:04.361272 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:04.361275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:04.361276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:04.361287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:04.363282 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:04.363692 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:04.363695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:04.363695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:04.363810 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:04.364037 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:04.364038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:04.364039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:04.364341 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:04.364521 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:04.364604 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:59:04.364705 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260619 20:59:04.365990 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:04.374938 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:04.367470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:43563 (local address 127.0.0.1:32819)
0619 20:59:04.367504 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:04.367506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:04.367641 (+ 135us) server_negotiation.cc:407] Connection header received
0619 20:59:04.367729 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:04.367730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:04.367741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:04.367768 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:04.368454 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:04.369213 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:04.370275 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:04.370615 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:04.370655 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:04.372029 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:04.372030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:04.372031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:04.372037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:04.373565 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:04.374049 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:04.374050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:04.374050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:04.374102 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:04.374356 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:04.374357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:04.374357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:04.374481 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:04.374552 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:04.374777 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 20:59:04.374852 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:59:04.387259 21326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:04.387945 21326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:04.392490 21326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04a23a6d959c435eb9e5301102cb1719: Generated new TSK 0
I20260619 20:59:04.392634 21326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:04.395432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21237
W20260619 20:59:04.399549 21222 connection.cc:535] client connection to 127.0.0.1:32819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32819): Connection reset by peer (error 104)
2026-06-19T20:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21336](info): set up 2 sockets
Jun 19 20:59:04 dist-test-slave-8bgz krb5kdc[21336](info): commencing operation
krb5kdc: starting...
W20260619 20:59:05.422189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:05 dist-test-slave-8bgz krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:05.532997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:05.612362 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:05.612589 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:05.612632 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:05.613875 21349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:05.613943 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:05.613970 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:05.614002 21349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:05.614035 21349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:05.615449 21349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:05.615708 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:05.615943 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:05.620652 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:05.620682 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:05.620738 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:05.621057 21349 server_base.cc:1034] running on GCE node
I20260619 20:59:05.621299 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:05.621572 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:05.622730 21349 hybrid_clock.cc:648] HybridClock initialized: now 1781902745622719 us; error 42 us; skew 500 ppm
Jun 19 20:59:05 dist-test-slave-8bgz krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:05.625156 21349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:05.625566 21349 webserver.cc:460] Webserver started at http://127.0.0.1:39921/ using document root <none> and password file <none>
I20260619 20:59:05.625728 21349 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:05.625764 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:05.625847 21349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:05.626847 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "d34684e8052b4ce3bff148d7e1a712d5"
format_stamp: "Formatted at 2026-06-19 20:59:05 on dist-test-slave-8bgz"
server_key: "df4ff14a7d8e3a4b68be3eb9473b6d7a"
server_key_iv: "eeed07295deac5e475f00d3166813c59"
server_key_version: "encryptionkey@0"
I20260619 20:59:05.627143 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "d34684e8052b4ce3bff148d7e1a712d5"
format_stamp: "Formatted at 2026-06-19 20:59:05 on dist-test-slave-8bgz"
server_key: "df4ff14a7d8e3a4b68be3eb9473b6d7a"
server_key_iv: "eeed07295deac5e475f00d3166813c59"
server_key_version: "encryptionkey@0"
I20260619 20:59:05.628276 21349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:59:05.628980 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:05.629175 21349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:05.629266 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "d34684e8052b4ce3bff148d7e1a712d5"
format_stamp: "Formatted at 2026-06-19 20:59:05 on dist-test-slave-8bgz"
server_key: "df4ff14a7d8e3a4b68be3eb9473b6d7a"
server_key_iv: "eeed07295deac5e475f00d3166813c59"
server_key_version: "encryptionkey@0"
I20260619 20:59:05.629346 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:05.648435 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:05.650121 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:05.650291 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:05.654023 21349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44307
I20260619 20:59:05.654075 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44307 every 8 connection(s)
I20260619 20:59:05.654407 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:05.655223 21417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:05.657022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21349
I20260619 20:59:05.657131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:05.657315 392 external_mini_cluster.cc:1428] Setting key f565db6057a41061429414936d114750
I20260619 20:59:05.657639 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5: Bootstrap starting.
I20260619 20:59:05.658298 21417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:05.658581 21417 log.cc:824] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:05.659228 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5: No bootstrap required, opened a new log
I20260619 20:59:05.660966 21417 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:59:05.661092 21417 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:05.661114 21417 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34684e8052b4ce3bff148d7e1a712d5, State: Initialized, Role: FOLLOWER
I20260619 20:59:05.661248 21417 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:59:05.661334 21417 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:05.661391 21417 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:05.661458 21417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:05.662083 21417 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:59:05.662201 21417 leader_election.cc:302] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34684e8052b4ce3bff148d7e1a712d5; no voters:
I20260619 20:59:05.662420 21417 leader_election.cc:288] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:05.662446 21422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:05.662559 21422 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [term 1 LEADER]: Becoming Leader. State: Replica: d34684e8052b4ce3bff148d7e1a712d5, State: Running, Role: LEADER
I20260619 20:59:05.662672 21417 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:05.662688 21422 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } }
I20260619 20:59:05.663064 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:05.657795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:33147 (local address 127.0.0.1:44307)
0619 20:59:05.657972 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:05.657982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:05.658016 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:59:05.658562 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:05.658570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:05.658653 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:05.658781 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:05.659027 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:05.659967 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:05.660751 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:05.661232 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:05.661326 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:05.662012 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:05.662016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:05.662016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:05.662181 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:05.662255 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:05.662270 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:05.662397 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:05.662605 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 20:59:05.662633 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260619 20:59:05.663599 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34684e8052b4ce3bff148d7e1a712d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } } }
I20260619 20:59:05.663738 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:05.663995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:05.664187 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:05.664311 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34684e8052b4ce3bff148d7e1a712d5. Latest consensus state: current_term: 1 leader_uuid: "d34684e8052b4ce3bff148d7e1a712d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34684e8052b4ce3bff148d7e1a712d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44307 } } }
I20260619 20:59:05.665516 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:05.665920 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:05.667529 21432 catalog_manager.cc:1347] Generated new cluster ID: f465b732c47b41f2a19a4e816ffe158d
I20260619 20:59:05.667575 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:05.668497 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:05.665386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:33471 (local address 127.0.0.1:44307)
0619 20:59:05.665420 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:05.665422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:05.665532 (+ 110us) server_negotiation.cc:407] Connection header received
0619 20:59:05.665601 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:05.665602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:05.665612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:05.665655 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:05.665793 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:05.666495 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:05.667359 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:05.667577 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:05.667651 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:05.668278 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:05.668279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:05.668280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:05.668284 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:05.668295 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:05.668299 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:05.668364 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:05.668422 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:59:05.668429 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:59:05.674794 21432 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:05.675460 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:05.682621 21432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34684e8052b4ce3bff148d7e1a712d5: Generated new TSK 0
I20260619 20:59:05.682772 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:05.689891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21349
W20260619 20:59:05.693715 21334 connection.cc:535] client connection to 127.0.0.1:44307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44307): Connection reset by peer (error 104)
2026-06-19T20:59:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:05 dist-test-slave-8bgz krb5kdc[21448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:05 dist-test-slave-8bgz krb5kdc[21448](info): set up 2 sockets
Jun 19 20:59:05 dist-test-slave-8bgz krb5kdc[21448](info): commencing operation
krb5kdc: starting...
W20260619 20:59:06.722360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:06 dist-test-slave-8bgz krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:06.840972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:06.919905 21461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:06.920133 21461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:06.920172 21461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:06.921303 21461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:06.921368 21461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:06.921393 21461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:06.921420 21461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:06.921451 21461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:06.922915 21461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:06.923175 21461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:06.923398 21461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:06.928063 21466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:06.928050 21469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:06.928081 21467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:06.928361 21461 server_base.cc:1034] running on GCE node
I20260619 20:59:06.928583 21461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:06.928846 21461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:06.929983 21461 hybrid_clock.cc:648] HybridClock initialized: now 1781902746929962 us; error 35 us; skew 500 ppm
Jun 19 20:59:06 dist-test-slave-8bgz krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:06.932507 21461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:06.932905 21461 webserver.cc:460] Webserver started at http://127.0.0.1:38789/ using document root <none> and password file <none>
I20260619 20:59:06.933120 21461 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:06.933183 21461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:06.933333 21461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:06.934422 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "38e1d97d37d348efa4731729bd5e2f9b"
format_stamp: "Formatted at 2026-06-19 20:59:06 on dist-test-slave-8bgz"
server_key: "53daf502aa7413265212a56c68267ce5"
server_key_iv: "82a2b8dd8c1fd349d81f43b4e1a905a8"
server_key_version: "encryptionkey@0"
I20260619 20:59:06.934768 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "38e1d97d37d348efa4731729bd5e2f9b"
format_stamp: "Formatted at 2026-06-19 20:59:06 on dist-test-slave-8bgz"
server_key: "53daf502aa7413265212a56c68267ce5"
server_key_iv: "82a2b8dd8c1fd349d81f43b4e1a905a8"
server_key_version: "encryptionkey@0"
I20260619 20:59:06.936100 21461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:06.936883 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:06.937067 21461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:06.937170 21461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "38e1d97d37d348efa4731729bd5e2f9b"
format_stamp: "Formatted at 2026-06-19 20:59:06 on dist-test-slave-8bgz"
server_key: "53daf502aa7413265212a56c68267ce5"
server_key_iv: "82a2b8dd8c1fd349d81f43b4e1a905a8"
server_key_version: "encryptionkey@0"
I20260619 20:59:06.937243 21461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:06.940572 21461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:06.942814 21461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:06.942972 21461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:06.946738 21461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41681
I20260619 20:59:06.946767 21528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41681 every 8 connection(s)
I20260619 20:59:06.947089 21461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:06.948010 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:06.950101 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b: Bootstrap starting.
I20260619 20:59:06.950690 21529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:06.950932 21529 log.cc:824] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:06.951565 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b: No bootstrap required, opened a new log
I20260619 20:59:06.953171 21529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260619 20:59:06.953310 21529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:06.953332 21529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e1d97d37d348efa4731729bd5e2f9b, State: Initialized, Role: FOLLOWER
I20260619 20:59:06.953428 21529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260619 20:59:06.953476 21529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:06.953496 21529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:06.953521 21529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:06.953974 21529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260619 20:59:06.954057 21529 leader_election.cc:302] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e1d97d37d348efa4731729bd5e2f9b; no voters:
I20260619 20:59:06.954200 21529 leader_election.cc:288] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:06.954250 21532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:06.954371 21532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [term 1 LEADER]: Becoming Leader. State: Replica: 38e1d97d37d348efa4731729bd5e2f9b, State: Running, Role: LEADER
I20260619 20:59:06.954452 21532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260619 20:59:06.954514 21529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:06.954914 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38e1d97d37d348efa4731729bd5e2f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } } }
I20260619 20:59:06.955087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21461
I20260619 20:59:06.955134 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:06.955225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:06.955071 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e1d97d37d348efa4731729bd5e2f9b. Latest consensus state: current_term: 1 leader_uuid: "38e1d97d37d348efa4731729bd5e2f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e1d97d37d348efa4731729bd5e2f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } } }
I20260619 20:59:06.955278 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:06.955394 21541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:06.955432 392 external_mini_cluster.cc:1428] Setting key 79f0df28805e390c78388f46420c56cf
I20260619 20:59:06.955922 21541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:06.957760 21541 catalog_manager.cc:1347] Generated new cluster ID: 67a8947b7d2445f5b453cdb5b23142f5
I20260619 20:59:06.957857 21541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:06.958038 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:06.955941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:46069 (local address 127.0.0.1:41681)
0619 20:59:06.956235 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:06.956245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:06.956277 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:59:06.956524 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:06.956532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:06.956564 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:06.956715 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:06.957472 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:06.957476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:06.957476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:06.957481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:06.957545 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:06.957561 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:06.957621 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:06.957827 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:59:06.957869 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260619 20:59:06.958932 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:06.961426 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:06.960412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:51869 (local address 127.0.0.1:41681)
0619 20:59:06.960444 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:06.960446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:06.960541 (+ 95us) server_negotiation.cc:407] Connection header received
0619 20:59:06.960593 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:06.960593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:06.960598 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:06.960667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:06.961216 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:06.961217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:06.961218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:06.961222 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:06.961234 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:06.961238 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:06.961307 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:06.961351 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:59:06.961364 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:59:06.972050 21541 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:06.972211 21541 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:06.979929 21541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38e1d97d37d348efa4731729bd5e2f9b: Generated new TSK 0
I20260619 20:59:06.980105 21541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:06.982628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21461
2026-06-19T20:59:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:07 dist-test-slave-8bgz krb5kdc[21560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:07 dist-test-slave-8bgz krb5kdc[21560](info): set up 2 sockets
Jun 19 20:59:07 dist-test-slave-8bgz krb5kdc[21560](info): commencing operation
krb5kdc: starting...
W20260619 20:59:08.018323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:08.130896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:08.209537 21573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:08.209802 21573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:08.209841 21573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:08.210968 21573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:08.211040 21573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:08.211062 21573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:08.211082 21573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:08.211109 21573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:08.212594 21573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:08.212872 21573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:08.213094 21573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:08.218150 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:08.218134 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:08.218353 21573 server_base.cc:1034] running on GCE node
W20260619 20:59:08.218123 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:08.218673 21573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:08.219031 21573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:08.220218 21573 hybrid_clock.cc:648] HybridClock initialized: now 1781902748220196 us; error 34 us; skew 500 ppm
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:08.225150 21573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:08.225629 21573 webserver.cc:460] Webserver started at http://127.0.0.1:44833/ using document root <none> and password file <none>
I20260619 20:59:08.225826 21573 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:08.225867 21573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:08.225967 21573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:08.227109 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "6cfb118b433c42768205f32921397a92"
format_stamp: "Formatted at 2026-06-19 20:59:08 on dist-test-slave-8bgz"
server_key: "590df97d10b33fb86a63e67e7af696a7"
server_key_iv: "f1790ae31080f7b28a4e47d93d3299d0"
server_key_version: "encryptionkey@0"
I20260619 20:59:08.227439 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "6cfb118b433c42768205f32921397a92"
format_stamp: "Formatted at 2026-06-19 20:59:08 on dist-test-slave-8bgz"
server_key: "590df97d10b33fb86a63e67e7af696a7"
server_key_iv: "f1790ae31080f7b28a4e47d93d3299d0"
server_key_version: "encryptionkey@0"
I20260619 20:59:08.228741 21573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:59:08.229489 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:08.229671 21573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:59:08.229763 21573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "6cfb118b433c42768205f32921397a92"
format_stamp: "Formatted at 2026-06-19 20:59:08 on dist-test-slave-8bgz"
server_key: "590df97d10b33fb86a63e67e7af696a7"
server_key_iv: "f1790ae31080f7b28a4e47d93d3299d0"
server_key_version: "encryptionkey@0"
I20260619 20:59:08.229822 21573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:08.236207 21573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:08.238512 21573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:08.238624 21573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:08.242656 21573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38177
I20260619 20:59:08.242703 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38177 every 8 connection(s)
I20260619 20:59:08.243026 21573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:08.243882 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:08.244982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21573
I20260619 20:59:08.245070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:08.245276 392 external_mini_cluster.cc:1428] Setting key 7327d3573a9915924049cc5450dcbc8d
I20260619 20:59:08.246651 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92: Bootstrap starting.
I20260619 20:59:08.247279 21641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:08.247587 21641 log.cc:824] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:08.248275 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92: No bootstrap required, opened a new log
I20260619 20:59:08.249977 21641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260619 20:59:08.250137 21641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:08.250175 21641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfb118b433c42768205f32921397a92, State: Initialized, Role: FOLLOWER
I20260619 20:59:08.250407 21641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260619 20:59:08.250517 21641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:08.250557 21641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:08.250599 21641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:08.251338 21641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260619 20:59:08.251444 21641 leader_election.cc:302] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cfb118b433c42768205f32921397a92; no voters:
I20260619 20:59:08.251618 21641 leader_election.cc:288] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:08.251641 21646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:08.251755 21646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [term 1 LEADER]: Becoming Leader. State: Replica: 6cfb118b433c42768205f32921397a92, State: Running, Role: LEADER
I20260619 20:59:08.251884 21641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:08.251892 21646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:59:08.252359 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cfb118b433c42768205f32921397a92. Latest consensus state: current_term: 1 leader_uuid: "6cfb118b433c42768205f32921397a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } } }
I20260619 20:59:08.252432 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:08.252632 21653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:08.252665 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cfb118b433c42768205f32921397a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfb118b433c42768205f32921397a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } } }
I20260619 20:59:08.252796 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:08.253113 21653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:08.255609 21653 catalog_manager.cc:1347] Generated new cluster ID: 675d44223c0348a6a20a979d8333b93b
I20260619 20:59:08.255661 21653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:08.256699 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:08.245798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:36495 (local address 127.0.0.1:38177)
0619 20:59:08.246046 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:08.246054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:08.246084 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:59:08.246643 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:08.246652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:08.246712 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:08.246854 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:08.247496 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:08.248488 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:08.249240 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:08.249669 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:08.249899 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:08.252879 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:08.252884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:08.252884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:08.252897 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:08.254811 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:08.255256 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:08.255260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:08.255260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:08.255388 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:08.255661 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:08.255662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:08.255662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:08.255987 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:08.256135 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:08.256399 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 20:59:08.256496 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20260619 20:59:08.257719 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:08.260471 21653 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:08.261111 21653 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:08.266139 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:08.259197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:48641 (local address 127.0.0.1:38177)
0619 20:59:08.259232 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:08.259233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:08.260194 (+ 961us) server_negotiation.cc:407] Connection header received
0619 20:59:08.260404 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:08.260405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:08.260415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:08.260462 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:08.261000 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:08.261486 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:08.262182 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:08.262436 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:08.262518 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:08.263653 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:08.263654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:08.263655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:08.263661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:08.265020 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:08.265371 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:08.265372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:08.265372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:08.265428 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:08.265640 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:08.265641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:08.265641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:08.265747 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:08.265857 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:08.265972 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 20:59:08.266029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:08.270735 21653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cfb118b433c42768205f32921397a92: Generated new TSK 0
I20260619 20:59:08.270944 21653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:08.275173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21573
W20260619 20:59:08.279390 21558 connection.cc:535] client connection to 127.0.0.1:38177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38177): Connection reset by peer (error 104)
2026-06-19T20:59:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21672](info): set up 2 sockets
Jun 19 20:59:08 dist-test-slave-8bgz krb5kdc[21672](info): commencing operation
krb5kdc: starting...
W20260619 20:59:09.306388 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:09 dist-test-slave-8bgz krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:09.417233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:09.497417 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:09.497608 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:09.497632 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:09.498766 21685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:09.498806 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:09.498821 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:09.498832 21685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:09.498844 21685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:09.500283 21685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:09.500510 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:09.500720 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:09.505591 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:09.505591 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:09.505584 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:09.505811 21685 server_base.cc:1034] running on GCE node
I20260619 20:59:09.506083 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:09.506410 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:09.507546 21685 hybrid_clock.cc:648] HybridClock initialized: now 1781902749507536 us; error 30 us; skew 500 ppm
Jun 19 20:59:09 dist-test-slave-8bgz krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:09.509968 21685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:09.510444 21685 webserver.cc:460] Webserver started at http://127.0.0.1:45693/ using document root <none> and password file <none>
I20260619 20:59:09.510671 21685 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:09.510735 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:09.510887 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:09.511907 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "9448c9a7d4284768aae7a38e8cd515e6"
format_stamp: "Formatted at 2026-06-19 20:59:09 on dist-test-slave-8bgz"
server_key: "bdc4ad32313fa69fdf5dfff721ca8099"
server_key_iv: "e921e2150dcdd8215478d29e6cd461dd"
server_key_version: "encryptionkey@0"
I20260619 20:59:09.512193 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "9448c9a7d4284768aae7a38e8cd515e6"
format_stamp: "Formatted at 2026-06-19 20:59:09 on dist-test-slave-8bgz"
server_key: "bdc4ad32313fa69fdf5dfff721ca8099"
server_key_iv: "e921e2150dcdd8215478d29e6cd461dd"
server_key_version: "encryptionkey@0"
I20260619 20:59:09.513414 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:59:09.514070 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:09.514276 21685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:09.514371 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "9448c9a7d4284768aae7a38e8cd515e6"
format_stamp: "Formatted at 2026-06-19 20:59:09 on dist-test-slave-8bgz"
server_key: "bdc4ad32313fa69fdf5dfff721ca8099"
server_key_iv: "e921e2150dcdd8215478d29e6cd461dd"
server_key_version: "encryptionkey@0"
I20260619 20:59:09.514457 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:09.535173 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:09.537078 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:09.537235 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:09.541481 21685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37781
I20260619 20:59:09.541555 21752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37781 every 8 connection(s)
I20260619 20:59:09.541869 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:09.542881 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:09.545164 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6: Bootstrap starting.
I20260619 20:59:09.545753 21753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:09.546041 21753 log.cc:824] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:09.546783 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6: No bootstrap required, opened a new log
I20260619 20:59:09.548426 21753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 20:59:09.548575 21753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:09.548636 21753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9448c9a7d4284768aae7a38e8cd515e6, State: Initialized, Role: FOLLOWER
I20260619 20:59:09.548830 21753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 20:59:09.548915 21753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:09.548967 21753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:09.549023 21753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:09.549559 21753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 20:59:09.549674 21753 leader_election.cc:302] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9448c9a7d4284768aae7a38e8cd515e6; no voters:
I20260619 20:59:09.549893 21753 leader_election.cc:288] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:09.549957 21756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:09.550117 21756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9448c9a7d4284768aae7a38e8cd515e6, State: Running, Role: LEADER
I20260619 20:59:09.550153 21753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:09.550288 21756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 20:59:09.550720 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9448c9a7d4284768aae7a38e8cd515e6. Latest consensus state: current_term: 1 leader_uuid: "9448c9a7d4284768aae7a38e8cd515e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 20:59:09.550846 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:09.550851 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9448c9a7d4284768aae7a38e8cd515e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9448c9a7d4284768aae7a38e8cd515e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 20:59:09.551054 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:09.551275 21765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:09.551535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21685
I20260619 20:59:09.551649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:09.551834 392 external_mini_cluster.cc:1428] Setting key 97ee87181b158cb5f577d5dd0be0aab3
I20260619 20:59:09.551831 21765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:09.553366 21765 catalog_manager.cc:1347] Generated new cluster ID: 9226c6ecc07e40c9840d9aad34567ac1
I20260619 20:59:09.553419 21765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:09.557040 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:09.552293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:57681 (local address 127.0.0.1:37781)
0619 20:59:09.552569 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:09.552578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:09.552607 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:59:09.552713 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:09.552719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:09.552780 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:09.552913 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:09.553182 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:09.554252 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:09.555079 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:09.555515 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:09.555650 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:09.556398 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:09.556402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:09.556402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:09.556587 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:09.556618 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:09.556629 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:09.556722 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:09.556830 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:59:09.556857 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260619 20:59:09.558110 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:09.565861 21765 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:09.566596 21765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:09.568460 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:09.559737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:45775 (local address 127.0.0.1:37781)
0619 20:59:09.559795 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:09.559797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:09.565617 (+ 5820us) server_negotiation.cc:407] Connection header received
0619 20:59:09.565641 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:09.565642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:09.565653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:09.565707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:09.565911 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:09.566518 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:09.567294 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:09.567486 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:09.567559 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:09.568206 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:09.568208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:09.568208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:09.568213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:09.568226 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:09.568230 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:09.568303 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:09.568368 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 20:59:09.568379 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":37}
I20260619 20:59:09.571588 21765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9448c9a7d4284768aae7a38e8cd515e6: Generated new TSK 0
I20260619 20:59:09.571790 21765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:09.575831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21685
W20260619 20:59:09.580020 21670 connection.cc:535] client connection to 127.0.0.1:37781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37781): Connection reset by peer (error 104)
2026-06-19T20:59:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:09 dist-test-slave-8bgz krb5kdc[21784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:09 dist-test-slave-8bgz krb5kdc[21784](info): set up 2 sockets
Jun 19 20:59:09 dist-test-slave-8bgz krb5kdc[21784](info): commencing operation
krb5kdc: starting...
W20260619 20:59:10.618399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:10 dist-test-slave-8bgz krb5kdc[21784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:59:10.728354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:10.805721 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:10.805922 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:10.805945 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:10.807019 21797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:10.807054 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:10.807068 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:10.807081 21797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:10.807093 21797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:10.808468 21797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:10.808635 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:10.808794 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:10.813472 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:10.813480 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:10.813482 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:10.813788 21797 server_base.cc:1034] running on GCE node
I20260619 20:59:10.814019 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:10.814355 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:10.815514 21797 hybrid_clock.cc:648] HybridClock initialized: now 1781902750815491 us; error 34 us; skew 500 ppm
Jun 19 20:59:10 dist-test-slave-8bgz krb5kdc[21784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:10.818027 21797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:59:10.818528 21797 webserver.cc:460] Webserver started at http://127.0.0.1:40779/ using document root <none> and password file <none>
I20260619 20:59:10.818750 21797 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:10.818817 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:10.818984 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:10.820004 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "a00428879231443782bfad0a3ccd08f1"
format_stamp: "Formatted at 2026-06-19 20:59:10 on dist-test-slave-8bgz"
server_key: "f699d6f7299ca045f9da4588193b8cdf"
server_key_iv: "881e0f704a9f42e9eef792a2fd3699f0"
server_key_version: "encryptionkey@0"
I20260619 20:59:10.820367 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "a00428879231443782bfad0a3ccd08f1"
format_stamp: "Formatted at 2026-06-19 20:59:10 on dist-test-slave-8bgz"
server_key: "f699d6f7299ca045f9da4588193b8cdf"
server_key_iv: "881e0f704a9f42e9eef792a2fd3699f0"
server_key_version: "encryptionkey@0"
I20260619 20:59:10.821653 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:59:10.822664 21812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:10.822834 21797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:59:10.822947 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "a00428879231443782bfad0a3ccd08f1"
format_stamp: "Formatted at 2026-06-19 20:59:10 on dist-test-slave-8bgz"
server_key: "f699d6f7299ca045f9da4588193b8cdf"
server_key_iv: "881e0f704a9f42e9eef792a2fd3699f0"
server_key_version: "encryptionkey@0"
I20260619 20:59:10.823036 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:10.827346 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:10.829533 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:10.829632 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:10.833283 21797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44379
I20260619 20:59:10.833369 21864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44379 every 8 connection(s)
I20260619 20:59:10.833642 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:10.834570 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:10.836551 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1: Bootstrap starting.
I20260619 20:59:10.837083 21865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:10.837369 21865 log.cc:824] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:10.838001 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1: No bootstrap required, opened a new log
I20260619 20:59:10.839548 21865 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260619 20:59:10.839691 21865 raft_consensus.cc:374] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:10.839735 21865 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00428879231443782bfad0a3ccd08f1, State: Initialized, Role: FOLLOWER
I20260619 20:59:10.839869 21865 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260619 20:59:10.839943 21865 raft_consensus.cc:388] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:10.839986 21865 raft_consensus.cc:482] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:10.840031 21865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:10.840516 21865 raft_consensus.cc:504] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260619 20:59:10.840622 21865 leader_election.cc:302] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a00428879231443782bfad0a3ccd08f1; no voters:
I20260619 20:59:10.840777 21865 leader_election.cc:288] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:10.840807 21868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:10.840996 21868 raft_consensus.cc:686] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [term 1 LEADER]: Becoming Leader. State: Replica: a00428879231443782bfad0a3ccd08f1, State: Running, Role: LEADER
I20260619 20:59:10.841043 21865 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:10.841171 21868 consensus_queue.cc:237] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260619 20:59:10.841753 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a00428879231443782bfad0a3ccd08f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260619 20:59:10.841763 21870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a00428879231443782bfad0a3ccd08f1. Latest consensus state: current_term: 1 leader_uuid: "a00428879231443782bfad0a3ccd08f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00428879231443782bfad0a3ccd08f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260619 20:59:10.841845 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:10.841861 21870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:10.842316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21797
I20260619 20:59:10.842056 21877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:10.842429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:10.842625 392 external_mini_cluster.cc:1428] Setting key dcb3fcdd03b68a6fd3f06fa23311a6f5
I20260619 20:59:10.842856 21877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:10.844488 21877 catalog_manager.cc:1347] Generated new cluster ID: 310122ff9b0a4c2c86af40e2b24199d5
I20260619 20:59:10.844550 21877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:10.845193 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:10.843229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:35979 (local address 127.0.0.1:44379)
0619 20:59:10.843414 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:10.843423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:10.843467 (+ 44us) server_negotiation.cc:407] Connection header received
0619 20:59:10.843633 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:10.843640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:10.843667 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:10.843797 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:10.844782 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:10.844785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:10.844785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:10.844789 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:10.844829 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:10.844842 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:10.844904 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:10.844994 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:59:10.845031 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260619 20:59:10.845963 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:10.848393 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:10.847357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:37407 (local address 127.0.0.1:44379)
0619 20:59:10.847386 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:10.847387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:10.847512 (+ 125us) server_negotiation.cc:407] Connection header received
0619 20:59:10.847575 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:10.847575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:10.847580 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:10.847647 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:10.848183 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:10.848184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:10.848184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:10.848188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:10.848199 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:10.848203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:10.848265 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:10.848322 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 20:59:10.848334 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:59:10.870160 21877 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:10.870380 21877 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:10.877321 21877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a00428879231443782bfad0a3ccd08f1: Generated new TSK 0
I20260619 20:59:10.877497 21877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:10.886163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21797
2026-06-19T20:59:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:10 dist-test-slave-8bgz krb5kdc[21896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:10 dist-test-slave-8bgz krb5kdc[21896](info): set up 2 sockets
Jun 19 20:59:10 dist-test-slave-8bgz krb5kdc[21896](info): commencing operation
krb5kdc: starting...
W20260619 20:59:11.906466 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:11 dist-test-slave-8bgz krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:12.016789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:12.096746 21909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:12.096988 21909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:12.097028 21909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:12.098182 21909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:12.098300 21909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:12.098330 21909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:12.098357 21909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:12.098387 21909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:12.099853 21909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:12.100123 21909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:12.100348 21909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:12.105103 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:12.105144 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:12.105079 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:12.105355 21909 server_base.cc:1034] running on GCE node
I20260619 20:59:12.105700 21909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:12.105978 21909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:12.107134 21909 hybrid_clock.cc:648] HybridClock initialized: now 1781902752107094 us; error 58 us; skew 500 ppm
Jun 19 20:59:12 dist-test-slave-8bgz krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:12.109552 21909 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:12.109972 21909 webserver.cc:460] Webserver started at http://127.0.98.62:44231/ using document root <none> and password file <none>
I20260619 20:59:12.110181 21909 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:12.110270 21909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:12.110442 21909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:12.111425 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "39b3ad9982ff457490ca3911a9bc6a69"
format_stamp: "Formatted at 2026-06-19 20:59:12 on dist-test-slave-8bgz"
server_key: "5a5ab472a56e5c38cefd498badaf8984"
server_key_iv: "5a8c164ceb599e446d9a032ac16b9e71"
server_key_version: "encryptionkey@0"
I20260619 20:59:12.111778 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "39b3ad9982ff457490ca3911a9bc6a69"
format_stamp: "Formatted at 2026-06-19 20:59:12 on dist-test-slave-8bgz"
server_key: "5a5ab472a56e5c38cefd498badaf8984"
server_key_iv: "5a8c164ceb599e446d9a032ac16b9e71"
server_key_version: "encryptionkey@0"
I20260619 20:59:12.113047 21909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:59:12.113744 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:12.113924 21909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:12.114018 21909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "39b3ad9982ff457490ca3911a9bc6a69"
format_stamp: "Formatted at 2026-06-19 20:59:12 on dist-test-slave-8bgz"
server_key: "5a5ab472a56e5c38cefd498badaf8984"
server_key_iv: "5a8c164ceb599e446d9a032ac16b9e71"
server_key_version: "encryptionkey@0"
I20260619 20:59:12.114101 21909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:12.129258 21909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:12.131186 21909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:12.131367 21909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:12.135118 21909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41077
I20260619 20:59:12.135167 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41077 every 8 connection(s)
I20260619 20:59:12.135509 21909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:12.136329 21977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:12.138556 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69: Bootstrap starting.
I20260619 20:59:12.139129 21977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:12.139412 21977 log.cc:824] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:12.140053 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69: No bootstrap required, opened a new log
I20260619 20:59:12.140939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 21909
I20260619 20:59:12.141065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:12.141237 392 external_mini_cluster.cc:1428] Setting key 70709e588f447612e4d763a18785a3ae
I20260619 20:59:12.142092 21977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } }
I20260619 20:59:12.142292 21977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:12.142385 21977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b3ad9982ff457490ca3911a9bc6a69, State: Initialized, Role: FOLLOWER
I20260619 20:59:12.142517 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } }
I20260619 20:59:12.142597 21977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:12.142619 21977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:12.142688 21977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:12.143724 21977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } }
I20260619 20:59:12.143831 21977 leader_election.cc:302] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39b3ad9982ff457490ca3911a9bc6a69; no voters:
I20260619 20:59:12.143983 21977 leader_election.cc:288] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:12.144042 21982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:12.144207 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:12.144205 21982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [term 1 LEADER]: Becoming Leader. State: Replica: 39b3ad9982ff457490ca3911a9bc6a69, State: Running, Role: LEADER
I20260619 20:59:12.144359 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } }
I20260619 20:59:12.144668 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39b3ad9982ff457490ca3911a9bc6a69. Latest consensus state: current_term: 1 leader_uuid: "39b3ad9982ff457490ca3911a9bc6a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } } }
I20260619 20:59:12.144804 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39b3ad9982ff457490ca3911a9bc6a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b3ad9982ff457490ca3911a9bc6a69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41077 } } }
I20260619 20:59:12.144881 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:12.145097 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:12.145287 21991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:12.145829 21991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:12.147399 21991 catalog_manager.cc:1347] Generated new cluster ID: 91b8bf860831414981e66e8c82d1019d
I20260619 20:59:12.147454 21991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:59:12 dist-test-slave-8bgz krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:12.152107 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:12.141728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36135 (local address 127.0.98.62:41077)
0619 20:59:12.141949 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:12.141960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:12.141995 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:59:12.142114 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:12.142122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:12.142173 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:12.142334 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:12.143094 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:12.144549 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:12.145639 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:12.146200 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:12.146348 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:12.148927 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:12.148930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:12.148931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:12.148942 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:12.150643 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:12.151020 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:12.151023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:12.151024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:12.151126 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:12.151363 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:12.151364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:12.151365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:12.151581 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:12.151753 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:12.151866 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 20:59:12.151965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260619 20:59:12.153007 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:12.164242 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:12.154325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60271 (local address 127.0.98.62:41077)
0619 20:59:12.154365 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:12.154366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:12.158387 (+ 4021us) server_negotiation.cc:407] Connection header received
0619 20:59:12.158457 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:12.158458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:12.158468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:12.158521 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:12.159034 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:12.159512 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:12.160190 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:12.160428 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:12.160500 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:12.161617 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:12.161619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:12.161619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:12.161626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:12.163080 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:12.163447 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:12.163448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:12.163448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:12.163501 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:12.163725 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:12.163726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:12.163726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:12.163836 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:12.163944 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:12.164071 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:59:12.164131 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:59:12.164641 21991 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:12.165151 21991 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:12.169744 21991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39b3ad9982ff457490ca3911a9bc6a69: Generated new TSK 0
I20260619 20:59:12.169921 21991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:12.174510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 21909
W20260619 20:59:12.178277 21894 connection.cc:535] client connection to 127.0.98.62:41077 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41077): Connection reset by peer (error 104)
2026-06-19T20:59:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:12 dist-test-slave-8bgz krb5kdc[22008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:12 dist-test-slave-8bgz krb5kdc[22008](info): set up 2 sockets
Jun 19 20:59:12 dist-test-slave-8bgz krb5kdc[22008](info): commencing operation
krb5kdc: starting...
W20260619 20:59:13.206341 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:13 dist-test-slave-8bgz krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:13.319113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:13.398244 22021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:13.398468 22021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:13.398506 22021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:13.399657 22021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:13.399729 22021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:13.399752 22021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:13.399775 22021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:13.399803 22021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:13.401228 22021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:13.401497 22021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:13.401726 22021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:13.406512 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:13.406530 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:13.406793 22021 server_base.cc:1034] running on GCE node
W20260619 20:59:13.406555 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:13.407069 22021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:13.407327 22021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:13.408519 22021 hybrid_clock.cc:648] HybridClock initialized: now 1781902753408484 us; error 29 us; skew 500 ppm
Jun 19 20:59:13 dist-test-slave-8bgz krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:13.415692 22021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:13.416131 22021 webserver.cc:460] Webserver started at http://127.0.98.62:34641/ using document root <none> and password file <none>
I20260619 20:59:13.416357 22021 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:13.416424 22021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:13.416589 22021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:13.417634 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "8aa0990e164e41d58588e81d05db5e62"
format_stamp: "Formatted at 2026-06-19 20:59:13 on dist-test-slave-8bgz"
server_key: "265106daa63ccbaf7fc79459914ed057"
server_key_iv: "de4b5e1dd0dfacbedd0290df0dcf4056"
server_key_version: "encryptionkey@0"
I20260619 20:59:13.417999 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "8aa0990e164e41d58588e81d05db5e62"
format_stamp: "Formatted at 2026-06-19 20:59:13 on dist-test-slave-8bgz"
server_key: "265106daa63ccbaf7fc79459914ed057"
server_key_iv: "de4b5e1dd0dfacbedd0290df0dcf4056"
server_key_version: "encryptionkey@0"
I20260619 20:59:13.419373 22021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:59:13.420087 22036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:13.420281 22021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:13.420374 22021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "8aa0990e164e41d58588e81d05db5e62"
format_stamp: "Formatted at 2026-06-19 20:59:13 on dist-test-slave-8bgz"
server_key: "265106daa63ccbaf7fc79459914ed057"
server_key_iv: "de4b5e1dd0dfacbedd0290df0dcf4056"
server_key_version: "encryptionkey@0"
I20260619 20:59:13.420435 22021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:13.431516 22021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:13.433281 22021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:13.433386 22021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:13.436980 22021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40871
I20260619 20:59:13.437027 22088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40871 every 8 connection(s)
I20260619 20:59:13.437363 22021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:13.438189 22089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:13.440413 22089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62: Bootstrap starting.
I20260619 20:59:13.440917 22089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:13.441134 22089 log.cc:824] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:13.441738 22089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62: No bootstrap required, opened a new log
I20260619 20:59:13.443450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22021
I20260619 20:59:13.443567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:13.443768 392 external_mini_cluster.cc:1428] Setting key 0c7b2cf08c16e18555edbe73bb64fa7d
I20260619 20:59:13.443678 22089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } }
I20260619 20:59:13.443835 22089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:13.443890 22089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa0990e164e41d58588e81d05db5e62, State: Initialized, Role: FOLLOWER
I20260619 20:59:13.444039 22089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } }
I20260619 20:59:13.444123 22089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:13.444224 22089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:13.444312 22089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:13.444993 22089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } }
I20260619 20:59:13.445103 22089 leader_election.cc:302] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aa0990e164e41d58588e81d05db5e62; no voters:
I20260619 20:59:13.445259 22089 leader_election.cc:288] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:13.445281 22094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:13.445427 22094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [term 1 LEADER]: Becoming Leader. State: Replica: 8aa0990e164e41d58588e81d05db5e62, State: Running, Role: LEADER
I20260619 20:59:13.445556 22089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:13.445570 22094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } }
I20260619 20:59:13.445974 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aa0990e164e41d58588e81d05db5e62. Latest consensus state: current_term: 1 leader_uuid: "8aa0990e164e41d58588e81d05db5e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } } }
I20260619 20:59:13.446184 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:13.445976 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aa0990e164e41d58588e81d05db5e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aa0990e164e41d58588e81d05db5e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40871 } } }
I20260619 20:59:13.446465 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:13.446630 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:13.447152 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:13.448681 22103 catalog_manager.cc:1347] Generated new cluster ID: cc94f7348ad7448e88bbfcd37491c70b
I20260619 20:59:13.448731 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:13.449888 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:13.444259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50363 (local address 127.0.98.62:40871)
0619 20:59:13.444450 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:13.444459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:13.444493 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:59:13.444895 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:13.444903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:13.444961 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:13.445116 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:13.445351 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:13.446913 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:13.447796 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:13.448398 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:13.448504 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:13.449222 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:13.449225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:13.449225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:13.449416 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:13.449449 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:13.449462 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:13.449550 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:13.449677 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:59:13.449701 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260619 20:59:13.451102 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:13.457960 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:13.454506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60193 (local address 127.0.98.62:40871)
0619 20:59:13.454539 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:13.454541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:13.454660 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:59:13.454722 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:13.454723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:13.454734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:13.454786 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:13.454926 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:13.455671 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:13.456643 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:13.456974 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:13.457050 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:13.457705 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:13.457706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:13.457707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:13.457711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:13.457727 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:13.457732 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:13.457795 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:13.457867 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 20:59:13.457879 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:13.485986 22103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:13.486605 22103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:13.493299 22103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aa0990e164e41d58588e81d05db5e62: Generated new TSK 0
I20260619 20:59:13.493500 22103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:13.500710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22021
W20260619 20:59:13.505043 22006 connection.cc:535] client connection to 127.0.98.62:40871 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40871): Connection reset by peer (error 104)
2026-06-19T20:59:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:13 dist-test-slave-8bgz krb5kdc[22120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:13 dist-test-slave-8bgz krb5kdc[22120](info): set up 2 sockets
Jun 19 20:59:13 dist-test-slave-8bgz krb5kdc[22120](info): commencing operation
krb5kdc: starting...
W20260619 20:59:14.534492 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:14 dist-test-slave-8bgz krb5kdc[22120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:14.647266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:14.725855 22133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:14.726080 22133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:14.726118 22133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:14.727247 22133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:14.727316 22133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:14.727347 22133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:14.727376 22133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:14.727406 22133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:14.728835 22133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:14.729104 22133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:14.729346 22133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:14.733824 22138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:14.733968 22139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:14.733834 22141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:14.734061 22133 server_base.cc:1034] running on GCE node
I20260619 20:59:14.734329 22133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:14.734604 22133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:14.735790 22133 hybrid_clock.cc:648] HybridClock initialized: now 1781902754735788 us; error 38 us; skew 500 ppm
Jun 19 20:59:14 dist-test-slave-8bgz krb5kdc[22120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:14.738160 22133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:14.738688 22133 webserver.cc:460] Webserver started at http://127.0.98.62:44155/ using document root <none> and password file <none>
I20260619 20:59:14.738940 22133 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:14.739007 22133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:14.739135 22133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:14.740116 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "7ab859445ef940199b7cc5b19b2a3897"
format_stamp: "Formatted at 2026-06-19 20:59:14 on dist-test-slave-8bgz"
server_key: "7d5cab40be99b615bae4078ad41bb618"
server_key_iv: "4912b0e52a1d60c4afc2f9f2961b1ebb"
server_key_version: "encryptionkey@0"
I20260619 20:59:14.740471 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ab859445ef940199b7cc5b19b2a3897"
format_stamp: "Formatted at 2026-06-19 20:59:14 on dist-test-slave-8bgz"
server_key: "7d5cab40be99b615bae4078ad41bb618"
server_key_iv: "4912b0e52a1d60c4afc2f9f2961b1ebb"
server_key_version: "encryptionkey@0"
I20260619 20:59:14.741741 22133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:59:14.742478 22148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:14.742661 22133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:14.742758 22133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "7ab859445ef940199b7cc5b19b2a3897"
format_stamp: "Formatted at 2026-06-19 20:59:14 on dist-test-slave-8bgz"
server_key: "7d5cab40be99b615bae4078ad41bb618"
server_key_iv: "4912b0e52a1d60c4afc2f9f2961b1ebb"
server_key_version: "encryptionkey@0"
I20260619 20:59:14.742839 22133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:14.745862 22133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:14.747824 22133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:14.747982 22133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:14.751989 22133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45503
I20260619 20:59:14.752027 22200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45503 every 8 connection(s)
I20260619 20:59:14.752319 22133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:14.753266 22201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:14.755478 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897: Bootstrap starting.
I20260619 20:59:14.755970 22201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:14.756197 22201 log.cc:824] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:14.756824 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897: No bootstrap required, opened a new log
I20260619 20:59:14.758417 22201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 20:59:14.758544 22201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:14.758566 22201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab859445ef940199b7cc5b19b2a3897, State: Initialized, Role: FOLLOWER
I20260619 20:59:14.758669 22201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 20:59:14.758721 22201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:14.758741 22201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:14.758766 22201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:14.759238 22201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 20:59:14.759317 22201 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab859445ef940199b7cc5b19b2a3897; no voters:
I20260619 20:59:14.759433 22201 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:14.759511 22204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:14.759670 22204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [term 1 LEADER]: Becoming Leader. State: Replica: 7ab859445ef940199b7cc5b19b2a3897, State: Running, Role: LEADER
I20260619 20:59:14.759680 22201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:14.759765 22204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 20:59:14.760320 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab859445ef940199b7cc5b19b2a3897. Latest consensus state: current_term: 1 leader_uuid: "7ab859445ef940199b7cc5b19b2a3897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260619 20:59:14.760412 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:14.760625 22213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:14.760571 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab859445ef940199b7cc5b19b2a3897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab859445ef940199b7cc5b19b2a3897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260619 20:59:14.760941 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:14.761029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22133
I20260619 20:59:14.761138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:14.761174 22213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:14.761325 392 external_mini_cluster.cc:1428] Setting key 5776816a94b39c3f90ce2da0fe319c32
I20260619 20:59:14.762718 22213 catalog_manager.cc:1347] Generated new cluster ID: ea61f23bae5a4542af5a0955fcb000f4
I20260619 20:59:14.762768 22213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:14.763955 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:14.761922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37639 (local address 127.0.98.62:45503)
0619 20:59:14.762125 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:14.762134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:14.762165 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:59:14.762318 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:14.762325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:14.762355 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:14.762954 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:14.763427 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:14.763430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:14.763430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:14.763433 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:14.763466 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:14.763479 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:14.763544 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:14.763643 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:59:14.763779 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260619 20:59:14.764837 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:14.767364 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:14.766298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33063 (local address 127.0.98.62:45503)
0619 20:59:14.766329 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:14.766331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:14.766445 (+ 114us) server_negotiation.cc:407] Connection header received
0619 20:59:14.766514 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:14.766515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:14.766519 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:14.766565 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:14.767120 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:14.767121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:14.767121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:14.767124 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:14.767136 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:14.767141 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:14.767209 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:14.767298 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:59:14.767309 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:59:14.787214 22213 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:14.787403 22213 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:14.793462 22213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab859445ef940199b7cc5b19b2a3897: Generated new TSK 0
I20260619 20:59:14.793655 22213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:14.796486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22133
2026-06-19T20:59:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:14 dist-test-slave-8bgz krb5kdc[22232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:14 dist-test-slave-8bgz krb5kdc[22232](info): set up 2 sockets
Jun 19 20:59:14 dist-test-slave-8bgz krb5kdc[22232](info): commencing operation
krb5kdc: starting...
W20260619 20:59:15.814414 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:15 dist-test-slave-8bgz krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:15.927194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:16.006631 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:16.006824 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:16.006847 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:16.007987 22245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:16.008028 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:16.008042 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:16.008054 22245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:16.008104 22245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:16.009493 22245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:16.009713 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:16.009944 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:16.014611 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:16.014729 22245 server_base.cc:1034] running on GCE node
W20260619 20:59:16.014611 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:16.014642 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:16.015100 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:16.015419 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:16.016598 22245 hybrid_clock.cc:648] HybridClock initialized: now 1781902756016576 us; error 34 us; skew 500 ppm
Jun 19 20:59:16 dist-test-slave-8bgz krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:16.019022 22245 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:16.019502 22245 webserver.cc:460] Webserver started at http://127.0.98.62:37927/ using document root <none> and password file <none>
I20260619 20:59:16.019734 22245 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:16.019801 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:16.019951 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:16.020953 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "6f2db95adb2a487195f5bafddc95aa87"
format_stamp: "Formatted at 2026-06-19 20:59:16 on dist-test-slave-8bgz"
server_key: "cb5a989c2940892ad083e75ff8cadd96"
server_key_iv: "7d94d7d218d7655c85ae25b1eaea80c8"
server_key_version: "encryptionkey@0"
I20260619 20:59:16.021306 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f2db95adb2a487195f5bafddc95aa87"
format_stamp: "Formatted at 2026-06-19 20:59:16 on dist-test-slave-8bgz"
server_key: "cb5a989c2940892ad083e75ff8cadd96"
server_key_iv: "7d94d7d218d7655c85ae25b1eaea80c8"
server_key_version: "encryptionkey@0"
I20260619 20:59:16.022630 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:59:16.023329 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:16.023495 22245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:59:16.023595 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "6f2db95adb2a487195f5bafddc95aa87"
format_stamp: "Formatted at 2026-06-19 20:59:16 on dist-test-slave-8bgz"
server_key: "cb5a989c2940892ad083e75ff8cadd96"
server_key_iv: "7d94d7d218d7655c85ae25b1eaea80c8"
server_key_version: "encryptionkey@0"
I20260619 20:59:16.023672 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:16.046350 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:16.048118 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:16.048296 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:16.052333 22245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37513
I20260619 20:59:16.052374 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37513 every 8 connection(s)
I20260619 20:59:16.052753 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:16.053550 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:16.055701 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87: Bootstrap starting.
I20260619 20:59:16.056237 22313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:16.056460 22313 log.cc:824] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:16.057071 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87: No bootstrap required, opened a new log
I20260619 20:59:16.058579 22313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } }
I20260619 20:59:16.058707 22313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:16.058729 22313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2db95adb2a487195f5bafddc95aa87, State: Initialized, Role: FOLLOWER
I20260619 20:59:16.058889 22313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } }
I20260619 20:59:16.058983 22313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:16.059006 22313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:16.059036 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:16.059484 22313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } }
I20260619 20:59:16.059566 22313 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2db95adb2a487195f5bafddc95aa87; no voters:
I20260619 20:59:16.059706 22313 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:16.059742 22316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:16.059857 22316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [term 1 LEADER]: Becoming Leader. State: Replica: 6f2db95adb2a487195f5bafddc95aa87, State: Running, Role: LEADER
I20260619 20:59:16.059968 22313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:16.059999 22316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } }
I20260619 20:59:16.060412 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2db95adb2a487195f5bafddc95aa87. Latest consensus state: current_term: 1 leader_uuid: "6f2db95adb2a487195f5bafddc95aa87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } } }
I20260619 20:59:16.060503 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:16.060544 22318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2db95adb2a487195f5bafddc95aa87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2db95adb2a487195f5bafddc95aa87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37513 } } }
I20260619 20:59:16.060633 22318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:16.061028 22323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:16.061249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22245
I20260619 20:59:16.061368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:16.061566 392 external_mini_cluster.cc:1428] Setting key e170b2b6036aa300faa9cd75d2e0f7bc
I20260619 20:59:16.061595 22323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:16.063180 22323 catalog_manager.cc:1347] Generated new cluster ID: 8e88b8bd54374d3795f1ad1abf0dd6e9
I20260619 20:59:16.063236 22323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:59:16 dist-test-slave-8bgz krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:16.077376 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:16.062040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60617 (local address 127.0.98.62:37513)
0619 20:59:16.066369 (+ 4329us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:16.066379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:16.066413 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:59:16.066528 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:16.066535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:16.066591 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:16.066736 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:16.067501 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:16.069304 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:16.070331 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:16.070942 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:16.071050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:16.073797 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:16.073801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:16.073801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:16.073812 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:16.075717 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:16.076121 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:16.076124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:16.076124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:16.076235 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:16.076449 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:16.076449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:16.076449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:16.076752 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:16.076880 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:16.077003 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:59:16.077127 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4270,"thread_start_us":66,"threads_started":1}
I20260619 20:59:16.078260 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:16.084416 22323 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:16.085175 22323 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:16.086525 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:16.079568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34619 (local address 127.0.98.62:37513)
0619 20:59:16.079603 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:16.079605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:16.079662 (+ 57us) server_negotiation.cc:407] Connection header received
0619 20:59:16.079724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:16.079725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:16.079734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:16.079773 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:16.080407 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:16.081092 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:16.082069 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:16.082394 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:16.082450 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:16.083799 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:16.083800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:16.083801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:16.083808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:16.085407 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:16.085766 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:16.085767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:16.085767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:16.085820 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:16.086013 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:16.086014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:16.086014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:16.086117 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:16.086240 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:16.086364 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:59:16.086441 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:59:16.089720 22323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2db95adb2a487195f5bafddc95aa87: Generated new TSK 0
I20260619 20:59:16.089886 22323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:16.092736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22245
2026-06-19T20:59:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:16 dist-test-slave-8bgz krb5kdc[22344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:16 dist-test-slave-8bgz krb5kdc[22344](info): set up 2 sockets
Jun 19 20:59:16 dist-test-slave-8bgz krb5kdc[22344](info): commencing operation
krb5kdc: starting...
W20260619 20:59:17.122640 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:17 dist-test-slave-8bgz krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:17.235301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:17.314150 22357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:17.314428 22357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:17.314452 22357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:17.315591 22357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:17.315634 22357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:17.315647 22357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:17.315659 22357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:17.315671 22357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:17.317137 22357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:17.317391 22357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:17.317610 22357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:17.322686 22365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:17.322670 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:17.322767 22357 server_base.cc:1034] running on GCE node
W20260619 20:59:17.322695 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:17.323105 22357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:17.323405 22357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:17.324546 22357 hybrid_clock.cc:648] HybridClock initialized: now 1781902757324528 us; error 36 us; skew 500 ppm
Jun 19 20:59:17 dist-test-slave-8bgz krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:17.326860 22357 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:17.327296 22357 webserver.cc:460] Webserver started at http://127.0.98.62:36619/ using document root <none> and password file <none>
I20260619 20:59:17.327473 22357 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:17.327512 22357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:17.327598 22357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:17.328564 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "b3301989872041a8ac8703ceb65c239b"
format_stamp: "Formatted at 2026-06-19 20:59:17 on dist-test-slave-8bgz"
server_key: "c70ff722d379d265d9cbc5a110aee6d5"
server_key_iv: "83353626617a7b6f23b74cb19b936f3a"
server_key_version: "encryptionkey@0"
I20260619 20:59:17.328874 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3301989872041a8ac8703ceb65c239b"
format_stamp: "Formatted at 2026-06-19 20:59:17 on dist-test-slave-8bgz"
server_key: "c70ff722d379d265d9cbc5a110aee6d5"
server_key_iv: "83353626617a7b6f23b74cb19b936f3a"
server_key_version: "encryptionkey@0"
I20260619 20:59:17.330305 22357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:59:17.331004 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:17.331157 22357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:17.331243 22357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "b3301989872041a8ac8703ceb65c239b"
format_stamp: "Formatted at 2026-06-19 20:59:17 on dist-test-slave-8bgz"
server_key: "c70ff722d379d265d9cbc5a110aee6d5"
server_key_iv: "83353626617a7b6f23b74cb19b936f3a"
server_key_version: "encryptionkey@0"
I20260619 20:59:17.331326 22357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:17.339373 22357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:17.341019 22357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:17.341122 22357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:17.344769 22357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33053
I20260619 20:59:17.344838 22424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33053 every 8 connection(s)
I20260619 20:59:17.345152 22357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:17.345957 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:17.348181 22425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b: Bootstrap starting.
I20260619 20:59:17.348740 22425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:17.349018 22425 log.cc:824] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:17.349453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22357
I20260619 20:59:17.349574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:17.349773 22425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b: No bootstrap required, opened a new log
I20260619 20:59:17.349782 392 external_mini_cluster.cc:1428] Setting key ed25dd08f953f84ff3e1ef8b3a84ccff
I20260619 20:59:17.351725 22425 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } }
I20260619 20:59:17.351868 22425 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:17.351908 22425 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3301989872041a8ac8703ceb65c239b, State: Initialized, Role: FOLLOWER
I20260619 20:59:17.352021 22425 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } }
I20260619 20:59:17.352092 22425 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:17.352113 22425 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:17.352138 22425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:17.352936 22425 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } }
I20260619 20:59:17.353020 22425 leader_election.cc:302] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3301989872041a8ac8703ceb65c239b; no voters:
I20260619 20:59:17.353140 22425 leader_election.cc:288] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:17.353223 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:17.353385 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [term 1 LEADER]: Becoming Leader. State: Replica: b3301989872041a8ac8703ceb65c239b, State: Running, Role: LEADER
I20260619 20:59:17.353415 22425 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:17.353475 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } }
I20260619 20:59:17.353890 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3301989872041a8ac8703ceb65c239b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } } }
I20260619 20:59:17.353926 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3301989872041a8ac8703ceb65c239b. Latest consensus state: current_term: 1 leader_uuid: "b3301989872041a8ac8703ceb65c239b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3301989872041a8ac8703ceb65c239b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33053 } } }
I20260619 20:59:17.354053 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:17.354542 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:17.355141 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:17.354897 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:17.356607 22439 catalog_manager.cc:1347] Generated new cluster ID: 21d0c7d30b5d4d159fcf3d5dd47d2f3e
I20260619 20:59:17.356667 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:17.359424 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:17.350340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45709 (local address 127.0.98.62:33053)
0619 20:59:17.350555 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:17.350565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:17.354469 (+ 3904us) server_negotiation.cc:407] Connection header received
0619 20:59:17.354637 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:17.354644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:17.354700 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:17.354859 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:17.355256 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:17.356229 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:17.357374 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:17.357761 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:17.357842 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:17.358568 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:17.358572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:17.358572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:17.358792 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:17.358828 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:17.358842 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:17.358914 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:17.359166 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 20:59:17.359228 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260619 20:59:17.360400 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:17.366428 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:17.362507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57189 (local address 127.0.98.62:33053)
0619 20:59:17.362543 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:17.362544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:17.362689 (+ 145us) server_negotiation.cc:407] Connection header received
0619 20:59:17.362798 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:17.362798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:17.362809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:17.362838 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:17.363015 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:17.363977 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:17.364987 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:17.365290 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:17.365328 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:17.366146 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:17.366147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:17.366148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:17.366154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:17.366167 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:17.366171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:17.366216 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:17.366332 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 20:59:17.366356 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:59:17.381023 22439 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:17.381534 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:17.384835 22439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3301989872041a8ac8703ceb65c239b: Generated new TSK 0
I20260619 20:59:17.385012 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:17.387918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22357
2026-06-19T20:59:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:17 dist-test-slave-8bgz krb5kdc[22456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:17 dist-test-slave-8bgz krb5kdc[22456](info): set up 2 sockets
Jun 19 20:59:17 dist-test-slave-8bgz krb5kdc[22456](info): commencing operation
krb5kdc: starting...
W20260619 20:59:18.454375 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:18 dist-test-slave-8bgz krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:18.568456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:18.646525 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:18.646719 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:18.646744 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:18.647914 22469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:18.647955 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:18.647970 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:18.647982 22469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:18.647994 22469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:18.649415 22469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:18.649667 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:18.649860 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:18.654609 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:18.654635 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:18.654644 22469 server_base.cc:1034] running on GCE node
W20260619 20:59:18.654616 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:18.655040 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:18.655364 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:18.656518 22469 hybrid_clock.cc:648] HybridClock initialized: now 1781902758656499 us; error 37 us; skew 500 ppm
Jun 19 20:59:18 dist-test-slave-8bgz krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:18.661793 22469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:18.662246 22469 webserver.cc:460] Webserver started at http://127.0.98.62:39003/ using document root <none> and password file <none>
I20260619 20:59:18.662487 22469 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:18.662531 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:18.662678 22469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:18.663686 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "2efad213e19f4a6d9c97af44e1f2164b"
format_stamp: "Formatted at 2026-06-19 20:59:18 on dist-test-slave-8bgz"
server_key: "e81ef0d791de19a373a95150c1d48e4a"
server_key_iv: "7418b124ad029b2b90d5b1ec8ce360db"
server_key_version: "encryptionkey@0"
I20260619 20:59:18.664034 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "2efad213e19f4a6d9c97af44e1f2164b"
format_stamp: "Formatted at 2026-06-19 20:59:18 on dist-test-slave-8bgz"
server_key: "e81ef0d791de19a373a95150c1d48e4a"
server_key_iv: "7418b124ad029b2b90d5b1ec8ce360db"
server_key_version: "encryptionkey@0"
I20260619 20:59:18.665336 22469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:59:18.666087 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:18.666307 22469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:18.666416 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "2efad213e19f4a6d9c97af44e1f2164b"
format_stamp: "Formatted at 2026-06-19 20:59:18 on dist-test-slave-8bgz"
server_key: "e81ef0d791de19a373a95150c1d48e4a"
server_key_iv: "7418b124ad029b2b90d5b1ec8ce360db"
server_key_version: "encryptionkey@0"
I20260619 20:59:18.666504 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:18.669595 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:18.671525 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:18.671700 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:18.675467 22469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39957
I20260619 20:59:18.675545 22536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39957 every 8 connection(s)
I20260619 20:59:18.675839 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:18.676792 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:18.678956 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b: Bootstrap starting.
I20260619 20:59:18.679447 22537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:18.679680 22537 log.cc:824] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:18.680277 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b: No bootstrap required, opened a new log
I20260619 20:59:18.681758 22537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260619 20:59:18.681882 22537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:59:18.681905 22537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efad213e19f4a6d9c97af44e1f2164b, State: Initialized, Role: FOLLOWER
I20260619 20:59:18.682040 22537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260619 20:59:18.682126 22537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:59:18.682178 22537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:59:18.682272 22537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:18.682605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22469
I20260619 20:59:18.682716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:18.682766 22537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260619 20:59:18.682924 22537 leader_election.cc:302] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efad213e19f4a6d9c97af44e1f2164b; no voters:
I20260619 20:59:18.682951 392 external_mini_cluster.cc:1428] Setting key c234dafdbbf4338959837b7aebfea460
I20260619 20:59:18.683097 22537 leader_election.cc:288] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:59:18.683121 22540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:18.683291 22540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [term 1 LEADER]: Becoming Leader. State: Replica: 2efad213e19f4a6d9c97af44e1f2164b, State: Running, Role: LEADER
I20260619 20:59:18.683357 22537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:18.683421 22540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } }
I20260619 20:59:18.684028 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2efad213e19f4a6d9c97af44e1f2164b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } } }
I20260619 20:59:18.684028 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efad213e19f4a6d9c97af44e1f2164b. Latest consensus state: current_term: 1 leader_uuid: "2efad213e19f4a6d9c97af44e1f2164b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efad213e19f4a6d9c97af44e1f2164b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39957 } } }
I20260619 20:59:18.684147 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:18.684154 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:18.684455 22550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:18.685290 22550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:18.685694 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:18.683590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33301 (local address 127.0.98.62:39957)
0619 20:59:18.683726 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:18.683733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:18.684174 (+ 441us) server_negotiation.cc:407] Connection header received
0619 20:59:18.684410 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:18.684417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:18.684449 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:18.684588 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:18.685267 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:18.685271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:18.685271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:18.685276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:18.685316 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:18.685330 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:18.685367 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:18.685490 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:59:18.685530 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":76,"threads_started":1}
I20260619 20:59:18.686856 22550 catalog_manager.cc:1347] Generated new cluster ID: 0439301005724d219be81419ec1952c7
I20260619 20:59:18.686851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:18.686908 22550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:18.689447 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:18.688308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54353 (local address 127.0.98.62:39957)
0619 20:59:18.688342 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:18.688344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:18.688452 (+ 108us) server_negotiation.cc:407] Connection header received
0619 20:59:18.688549 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:18.688549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:18.688553 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:18.688583 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:18.689205 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:18.689206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:18.689206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:59:18.689210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:18.689223 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:59:18.689227 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:18.689267 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:18.689374 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:59:18.689387 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:59:18.700139 22550 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:18.700304 22550 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:18.705003 22550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2efad213e19f4a6d9c97af44e1f2164b: Generated new TSK 0
I20260619 20:59:18.705164 22550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:59:18.707109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22469
2026-06-19T20:59:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1319 ms)
[----------] 18 tests from AuthTokenIssuingTest (23438 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:18 dist-test-slave-8bgz krb5kdc[22568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:18 dist-test-slave-8bgz krb5kdc[22568](info): set up 2 sockets
Jun 19 20:59:18 dist-test-slave-8bgz krb5kdc[22568](info): commencing operation
krb5kdc: starting...
W20260619 20:59:19.742394 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:19 dist-test-slave-8bgz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:19.856561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:19.938935 22581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:19.939138 22581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:19.939155 22581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:19.939174 22581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:19.940213 22581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:19.940245 22581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:19.940259 22581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:19.940271 22581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:19.940284 22581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:19.941640 22581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:19.941819 22581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:19.941977 22581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:19.946832 22589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:19.946779 22587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:19.946789 22586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:19.946813 22581 server_base.cc:1034] running on GCE node
I20260619 20:59:19.947253 22581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:19.947528 22581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:19.948668 22581 hybrid_clock.cc:648] HybridClock initialized: now 1781902759948654 us; error 30 us; skew 500 ppm
Jun 19 20:59:19 dist-test-slave-8bgz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:19.951063 22581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:19.951547 22581 webserver.cc:460] Webserver started at http://127.0.98.62:42909/ using document root <none> and password file <none>
I20260619 20:59:19.951785 22581 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:19.951853 22581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:19.952001 22581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:19.953037 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "5cfb1166e1904ae69d926d9b985b7cdf"
format_stamp: "Formatted at 2026-06-19 20:59:19 on dist-test-slave-8bgz"
server_key: "1f4ef332f827be4485754a3ea7225b40"
server_key_iv: "b2b0587726745f18fd8539241bcaf88a"
server_key_version: "encryptionkey@0"
I20260619 20:59:19.953382 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cfb1166e1904ae69d926d9b985b7cdf"
format_stamp: "Formatted at 2026-06-19 20:59:19 on dist-test-slave-8bgz"
server_key: "1f4ef332f827be4485754a3ea7225b40"
server_key_iv: "b2b0587726745f18fd8539241bcaf88a"
server_key_version: "encryptionkey@0"
I20260619 20:59:19.954715 22581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:59:19.955401 22596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:19.955574 22581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:19.955677 22581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "5cfb1166e1904ae69d926d9b985b7cdf"
format_stamp: "Formatted at 2026-06-19 20:59:19 on dist-test-slave-8bgz"
server_key: "1f4ef332f827be4485754a3ea7225b40"
server_key_iv: "b2b0587726745f18fd8539241bcaf88a"
server_key_version: "encryptionkey@0"
I20260619 20:59:19.955772 22581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:19.983057 22581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:19.984917 22581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:19.985080 22581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:19.989084 22581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35605
I20260619 20:59:19.989120 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35605 every 8 connection(s)
I20260619 20:59:19.989472 22581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:19.990424 22649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:19.991254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22581
I20260619 20:59:19.991360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:19.991555 392 external_mini_cluster.cc:1428] Setting key 3564d918d20d946eaf5f60148d08716a
I20260619 20:59:19.992516 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Jun 19 20:59:19 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 20:59:20.002643 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:19.993390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35605 (local address 127.0.0.1:56906)
0619 20:59:19.993549 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:19.993583 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:19.994082 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:19.994402 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:19.994405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:19.994682 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:19.994826 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:19.994830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:19.995839 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:19.995840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:19.996492 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:19.996494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:19.997015 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:19.997016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:19.997078 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:19.997603 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:19.997612 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:19.999071 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.000883 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.000890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.000892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.001092 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.001454 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.001455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.001455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.001571 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.002056 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.002063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.002130 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.002305 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.002461 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260619 20:59:20.002766 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:19.993579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:35605)
0619 20:59:19.993703 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:19.993707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:19.993737 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:59:19.994281 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:19.994283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:19.994299 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:19.994397 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:19.994923 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:19.995705 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:19.996594 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:19.996898 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:19.997030 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:19.999167 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:19.999174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:19.999174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:19.999181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.000761 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.001275 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.001279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.001279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.001376 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.001639 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.001640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.001640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.001963 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.002052 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.002474 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.002576 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260619 20:59:20.003603 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:20.004034 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.003884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46398)
0619 20:59:20.003940 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.003998 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:59:20.004240 22597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
I20260619 20:59:20.004743 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.004621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46408)
0619 20:59:20.004677 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.004710 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 20:59:20.004940 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44439: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:59:20.016959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44439
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:59:20.054180 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } attempt: 1
I20260619 20:59:20.054771 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.054633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46410)
0619 20:59:20.054697 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.054740 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:59:20.055270 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.055139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46424)
0619 20:59:20.055215 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.055245 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 20:59:20.055401 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44439: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
W20260619 20:59:20.099516 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:20.099726 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:20.099743 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:20.099763 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:20.100874 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:20.100915 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:20.100929 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:20.100941 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:20.100979 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:20.102509 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:20.102757 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:20.103006 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:20.107671 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:20.107697 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:20.107750 22656 server_base.cc:1034] running on GCE node
W20260619 20:59:20.107700 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:20.108094 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:20.108364 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:20.109499 22656 hybrid_clock.cc:648] HybridClock initialized: now 1781902760109485 us; error 34 us; skew 500 ppm
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:20.112126 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:59:20.112537 22656 webserver.cc:460] Webserver started at http://127.0.98.61:44001/ using document root <none> and password file <none>
I20260619 20:59:20.112763 22656 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:20.112830 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:20.112979 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:20.113976 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data/instance:
uuid: "69775c2c6ad8488ea26146758a57a5fc"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "5995ab19663ea5822ae38aaffd8a5a4d"
server_key_iv: "16c03be73fdb4a91214a2e5729bc9e46"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.114353 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal/instance:
uuid: "69775c2c6ad8488ea26146758a57a5fc"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "5995ab19663ea5822ae38aaffd8a5a4d"
server_key_iv: "16c03be73fdb4a91214a2e5729bc9e46"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.115643 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:20.116333 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:20.116501 22656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:20.116597 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "69775c2c6ad8488ea26146758a57a5fc"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "5995ab19663ea5822ae38aaffd8a5a4d"
server_key_iv: "16c03be73fdb4a91214a2e5729bc9e46"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.116676 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:20.123533 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } attempt: 2
I20260619 20:59:20.123908 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.123792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46430)
0619 20:59:20.123845 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.123880 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:59:20.124187 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.124121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46436)
0619 20:59:20.124146 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.124171 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 20:59:20.124281 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44439: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44439: connect: Connection refused (error 111)
I20260619 20:59:20.141379 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:20.143782 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:20.143947 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:20.149070 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44439
I20260619 20:59:20.149109 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44439 every 8 connection(s)
I20260619 20:59:20.149580 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:59:20.150858 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:20.151687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22656
I20260619 20:59:20.151801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-1/wal/instance
I20260619 20:59:20.152025 392 external_mini_cluster.cc:1428] Setting key 73bf81334c148fa800c9a085d7a07067
I20260619 20:59:20.155323 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:20.167955 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.156434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.98.62:35605)
0619 20:59:20.156482 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.156484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.156695 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:59:20.157206 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.157207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.157223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.157278 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.158282 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.159117 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.160517 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.160747 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.160834 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.164160 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.164161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.164162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.164169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.166163 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.166726 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.166728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.166729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.166793 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.167108 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.167109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.167110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.167259 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.167320 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.167716 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.167812 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:59:20.168545 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.156409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35605 (local address 127.0.0.1:56910)
0619 20:59:20.156556 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.156596 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.157082 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.157403 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.157409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.157856 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.158121 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.158128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.159268 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.159269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.160422 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.160425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.160858 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.160859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.161003 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.161557 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.161565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.164009 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.166287 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.166294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.166297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.166626 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.166872 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.166873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.166873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.167014 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.167359 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.167365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.167542 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.168061 (+ 519us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.168270 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260619 20:59:20.169162 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:20.177827 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.169455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46440)
0619 20:59:20.169661 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.169672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.169729 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.169863 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.169864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.170035 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.170108 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.170110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.171319 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.171320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.171817 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.171818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.172299 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.172300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.172361 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.172796 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.172801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.174940 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.176762 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.176763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.176763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.176919 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.177125 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.177126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.177126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.177207 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.177571 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.177571 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.177603 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.177663 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.177742 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 20:59:20.177953 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.169530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.61:44439)
0619 20:59:20.169676 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.169680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.169721 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:59:20.169788 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.169789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.169800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.169859 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.170176 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.171226 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.171912 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.172218 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.172304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.175027 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.175034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.175034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.175040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.176678 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.177016 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.177018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.177019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.177076 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.177293 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.177294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.177294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.177475 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.177550 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.177728 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.177809 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260619 20:59:20.182381 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:20.182710 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.182589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47626)
0619 20:59:20.182632 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.182685 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 20:59:20.182864 22672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260619 20:59:20.183287 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.183205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47630)
0619 20:59:20.183241 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.183267 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 20:59:20.183364 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:59:20.184675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32819
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:59:20.215421 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } attempt: 3
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:20.222667 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.215914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.0.98.61:44439)
0619 20:59:20.215982 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.215984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.215996 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:59:20.216046 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.216048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.216066 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.216134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.216454 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.216939 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.217648 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.217858 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.217932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.220133 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.220134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.220134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.220140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.221526 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.221955 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.221956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.221956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.222003 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.222190 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.222190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.222190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.222343 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.222395 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.222549 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.222605 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:59:20.222729 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.215847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46442)
0619 20:59:20.215912 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.215922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.215984 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.216143 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.216144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.216307 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.216370 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.216371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.217027 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.217029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.217549 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.217551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.217942 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.217943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.218018 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.218574 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.218579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.220011 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.221628 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.221629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.221630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.221869 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.222065 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.222066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.222066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.222133 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.222406 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.222407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.222457 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.222534 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.222634 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 20:59:20.223155 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:20.223519 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.223418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47642)
0619 20:59:20.223470 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.223500 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:59:20.223826 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.223745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47656)
0619 20:59:20.223789 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.223812 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:59:20.223927 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260619 20:59:20.232591 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 1
I20260619 20:59:20.232985 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.232857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47658)
0619 20:59:20.232909 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.232944 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:59:20.233306 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.233218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47664)
0619 20:59:20.233263 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.233288 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:59:20.233526 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260619 20:59:20.255056 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 1
I20260619 20:59:20.255501 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.255376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47680)
0619 20:59:20.255436 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.255474 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:59:20.255884 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.255800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47684)
0619 20:59:20.255844 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.255868 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:59:20.256009 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
W20260619 20:59:20.267584 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:20.267812 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:20.267858 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:20.267900 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:20.269037 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:20.269102 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:20.269157 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:20.269183 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:20.269205 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:20.270699 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35605,127.0.98.61:44439,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:20.270962 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:20.271175 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:20.275810 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:20.275818 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:20.275818 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:20.276270 22731 server_base.cc:1034] running on GCE node
I20260619 20:59:20.276496 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:20.276706 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:20.277886 22731 hybrid_clock.cc:648] HybridClock initialized: now 1781902760277869 us; error 29 us; skew 500 ppm
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:20.280452 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:59:20.280817 22731 webserver.cc:460] Webserver started at http://127.0.98.60:42273/ using document root <none> and password file <none>
I20260619 20:59:20.281000 22731 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:20.281039 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:20.281133 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:20.282083 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data/instance:
uuid: "5843d76488d243d6aed24a677b104bb1"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "42dd7dcbb995688d851c09cfc67e42e3"
server_key_iv: "cf6e9a03e77b1b698bc4b57408ec6fd4"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.282516 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal/instance:
uuid: "5843d76488d243d6aed24a677b104bb1"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "42dd7dcbb995688d851c09cfc67e42e3"
server_key_iv: "cf6e9a03e77b1b698bc4b57408ec6fd4"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.283718 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:20.284353 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:20.284512 22731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:20.284603 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "5843d76488d243d6aed24a677b104bb1"
format_stamp: "Formatted at 2026-06-19 20:59:20 on dist-test-slave-8bgz"
server_key: "42dd7dcbb995688d851c09cfc67e42e3"
server_key_iv: "cf6e9a03e77b1b698bc4b57408ec6fd4"
server_key_version: "encryptionkey@0"
I20260619 20:59:20.284682 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:20.301669 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 2
I20260619 20:59:20.302088 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.301960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47690)
0619 20:59:20.302019 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.302057 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:59:20.302385 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.302317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47694)
0619 20:59:20.302343 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.302367 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 20:59:20.302470 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260619 20:59:20.327626 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:20.329867 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:20.329977 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:20.331157 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 2
I20260619 20:59:20.331609 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.331468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47708)
0619 20:59:20.331536 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.331576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 20:59:20.331992 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.331904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47712)
0619 20:59:20.331940 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.331967 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 20:59:20.332094 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260619 20:59:20.334096 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32819
I20260619 20:59:20.334148 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32819 every 8 connection(s)
I20260619 20:59:20.334519 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:59:20.335441 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:20.336918 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:20.339766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22731
I20260619 20:59:20.339877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902497118594-392-0/minicluster-data/master-2/wal/instance
I20260619 20:59:20.340044 392 external_mini_cluster.cc:1428] Setting key 68f757e193bf42a7af3623e5ec5468c9
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:20.346403 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.337572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.98.62:35605)
0619 20:59:20.337636 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.337638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.337860 (+ 222us) server_negotiation.cc:407] Connection header received
0619 20:59:20.338360 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.338361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.338370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.338423 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.339005 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.339459 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.340835 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.341021 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.341098 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.343425 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.343426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.343427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.343434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.344932 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.345386 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.345387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.345387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.345434 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.345716 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.345717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.345717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.345817 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.345883 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.346254 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.346332 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:59:20.346583 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.337559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35605 (local address 127.0.0.1:56920)
0619 20:59:20.337747 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.337780 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.338255 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.338496 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.338500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.338727 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.338928 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.338932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.339533 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.339534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.340760 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.340761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.341117 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.341119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.341263 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.341731 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.341749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.343341 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.345026 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.345032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.345034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.345313 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.345518 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.345519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.345519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.345650 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.345902 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.345908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.346064 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.346153 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.346350 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260619 20:59:20.347088 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:20.349973 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.340435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.98.60:32819)
0619 20:59:20.340627 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.340631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.340681 (+ 50us) server_negotiation.cc:407] Connection header received
0619 20:59:20.340734 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.340737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.340756 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.340824 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.341437 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.342094 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.343152 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.343473 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.343548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.346025 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.346031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.346031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.346036 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.347945 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.348337 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.348339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.348339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.348388 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.349099 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.349099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.349099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.349223 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.349321 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.349477 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.349559 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:20.353856 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.347342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44439 (local address 127.0.0.1:46456)
0619 20:59:20.347390 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.347401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.347442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.347567 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.347568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.347708 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.347733 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.347734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.348570 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.348571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.349029 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.349030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.349402 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.349402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.349485 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.349894 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.349898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.351173 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.352814 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.352814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.352815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.353009 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.353196 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.353197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.353197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.353258 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.353549 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.353550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.353599 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.353681 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.353768 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:59:20.354147 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.347422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.98.61:44439)
0619 20:59:20.347465 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.347467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.347476 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:59:20.347511 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.347512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.347519 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.347567 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.347816 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.348503 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.349095 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.349322 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.349399 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.351277 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.351278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.351278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.351285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.352725 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.353085 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.353085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.353086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.353137 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.353318 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.353319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.353319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.353458 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.353533 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.353721 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.353807 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:59:20.354620 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:20.358147 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.350789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.0.98.62:35605)
0619 20:59:20.350845 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.350847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.350853 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:59:20.350871 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.350872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.350879 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.350931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.351391 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.352029 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.352985 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.353299 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.353359 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.355657 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.355659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.355659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.355666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.357061 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.357382 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.357383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.357384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.357433 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.357629 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.357630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.357631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.357741 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.357924 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.357960 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.358061 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:59:20.360952 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.354845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47728)
0619 20:59:20.354922 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.354931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.354981 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.355100 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.355101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.355247 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.355295 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.355298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.355914 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.355915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.356355 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.356356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.356781 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.356782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.356831 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.357186 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.357190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.358275 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.359959 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.359960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.359960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.360115 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.360318 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.360319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.360319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.360372 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.360621 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.360622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.360664 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.360741 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.360840 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 20:59:20.361213 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.354888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.0.98.60:32819)
0619 20:59:20.354918 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.354920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.354966 (+ 46us) server_negotiation.cc:407] Connection header received
0619 20:59:20.355034 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.355035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.355044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.355097 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.355365 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.355812 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.356410 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.356688 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.356780 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.358356 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.358357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.358357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.358364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.359869 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.360200 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.360201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.360201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.360248 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.360443 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.360444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.360444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.360547 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.360616 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.360775 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.360884 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":14}
I20260619 20:59:20.362490 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: Bootstrap starting.
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:20.363408 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:20.363705 22799 log.cc:824] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:20.364547 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: No bootstrap required, opened a new log
I20260619 20:59:20.365595 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.358782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.61:44439)
0619 20:59:20.358826 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.358828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.358836 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:59:20.358896 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.358897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.358906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.358960 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.359464 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.359958 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.360588 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.360808 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.360878 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.363120 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.363121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.363122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.363128 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.364642 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.364993 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.364993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.364994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.365039 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.365224 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.365225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.365225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.365315 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.365409 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.365495 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.365544 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:59:20.366752 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.366899 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5843d76488d243d6aed24a677b104bb1, State: Initialized, Role: FOLLOWER
I20260619 20:59:20.366997 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.367079 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:20.367318 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:20.367311 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.367408 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:20.368049 22623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5cfb1166e1904ae69d926d9b985b7cdf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934
W20260619 20:59:20.369027 22816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:20.369213 22816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:20.369715 22731 master_runner.cc:186] Error getting master registration for 127.0.98.62:35605: OK, instance_id { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" instance_seqno: 1781902759987926 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:59:20.372505 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:59:20.393605 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 3
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:20.400525 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.393897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47742)
0619 20:59:20.393955 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.393968 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.394015 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.394138 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.394139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.394359 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.394398 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.394400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.395036 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.395037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.395562 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.395564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.395953 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.395954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.396011 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.396463 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.396469 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.397807 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.399469 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.399470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.399470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.399645 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.399864 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.399865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.399865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.399921 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.400199 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.400199 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.400251 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.400336 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.400433 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:59:20.400525 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.393972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.98.60:32819)
0619 20:59:20.394036 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.394038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.394058 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:59:20.394081 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.394081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.394090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.394136 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.394479 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.394955 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.395650 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.395869 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.395952 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.397891 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.397893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.397893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.397900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.399366 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.399736 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.399737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.399738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.399789 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.399991 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.399992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.399993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.400128 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.400207 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.400360 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.400433 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:59:20.402074 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: Bootstrap starting.
I20260619 20:59:20.402766 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:20.403069 22724 log.cc:824] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:20.403769 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: No bootstrap required, opened a new log
I20260619 20:59:20.405755 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.405886 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69775c2c6ad8488ea26146758a57a5fc, State: Initialized, Role: FOLLOWER
I20260619 20:59:20.405968 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.406311 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:20.406255 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.406464 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:59:20.408092 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:20.408185 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:20.408234 22656 master_runner.cc:186] Error getting master registration for 127.0.98.62:35605: OK, instance_id { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" instance_seqno: 1781902759987926 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:59:20.410828 22656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:59:20.431286 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 3
Jun 19 20:59:20 dist-test-slave-8bgz krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:20.439546 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.431675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.98.60:32819)
0619 20:59:20.431748 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.431750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.431846 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:59:20.431873 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.431874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.431884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.431939 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.432319 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.433036 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.434020 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.434405 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.434484 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.436742 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.436743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.436743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.436750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.438373 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.438754 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.438756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.438756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.438830 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.439032 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.439033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.439034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.439169 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.439237 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.439385 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.439459 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:59:20.439545 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.431606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:47744)
0619 20:59:20.431725 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:20.431736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:20.431790 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:20.431967 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:20.431968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:20.432172 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:20.432220 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.432223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.433155 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.433156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.433921 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:20.433924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.434488 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.434489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:20.434558 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.435115 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:20.435121 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:20.436648 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:20.438447 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.438448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.438448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.438673 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.438902 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:20.438902 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:20.438903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:20.438967 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:20.439228 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:20.439228 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:20.439277 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:59:20.439347 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:59:20.439441 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 20:59:20.441538 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: Bootstrap starting.
I20260619 20:59:20.442329 22649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:20.442623 22649 log.cc:824] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:20.443426 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: No bootstrap required, opened a new log
I20260619 20:59:20.444803 22649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.444940 22649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfb1166e1904ae69d926d9b985b7cdf, State: Initialized, Role: FOLLOWER
I20260619 20:59:20.445017 22649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.445303 22649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:20.445371 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:59:20.445437 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:20.446193 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.446535 22623 leader_election.cc:288] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [CANDIDATE]: Term 1 election: Requested vote from peers 69775c2c6ad8488ea26146758a57a5fc (127.0.98.61:44439), 5843d76488d243d6aed24a677b104bb1 (127.0.98.60:32819)
I20260619 20:59:20.445317 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.447057 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:59:20.447474 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 20:59:20.447085 22698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69775c2c6ad8488ea26146758a57a5fc"
I20260619 20:59:20.447649 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [term 0 FOLLOWER]: Advancing to term 1
W20260619 20:59:20.447692 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:20.448539 22698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfb1166e1904ae69d926d9b985b7cdf in term 1.
I20260619 20:59:20.448764 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5843d76488d243d6aed24a677b104bb1"
I20260619 20:59:20.448889 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:20.449697 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfb1166e1904ae69d926d9b985b7cdf in term 1.
I20260619 20:59:20.450663 22597 leader_election.cc:302] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfb1166e1904ae69d926d9b985b7cdf, 69775c2c6ad8488ea26146758a57a5fc; no voters:
I20260619 20:59:20.450786 22833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:20.450945 22833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [term 1 LEADER]: Becoming Leader. State: Replica: 5cfb1166e1904ae69d926d9b985b7cdf, State: Running, Role: LEADER
I20260619 20:59:20.451033 22833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260619 20:59:20.451465 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfb1166e1904ae69d926d9b985b7cdf. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.451555 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:20.453207 22581 master_runner.cc:263] No leader master found from master 5cfb1166e1904ae69d926d9b985b7cdf
I20260619 20:59:20.453258 22581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:59:20.458336 22849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:20.458547 22849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:20.465023 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.453529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.0.98.62:35605)
0619 20:59:20.453566 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.453568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.454421 (+ 853us) server_negotiation.cc:407] Connection header received
0619 20:59:20.454533 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.454534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.454543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.454571 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.455422 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.456171 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.457271 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.457560 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.457598 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.462133 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.462135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.462135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.462142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.463819 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.464190 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.464192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.464192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.464246 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.464455 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.464455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.464456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.464574 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.464656 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.464849 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.464924 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:20.469470 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.454150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.0.98.60:32819)
0619 20:59:20.454186 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.454187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.462349 (+ 8162us) server_negotiation.cc:407] Connection header received
0619 20:59:20.462441 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.462442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.462452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.462480 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.463127 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.463846 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.464834 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.465137 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.465184 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.466681 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.466682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.466682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.466689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.468275 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.468674 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.468675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.468675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.468728 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.468920 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.468921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.468921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.469035 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.469100 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.469293 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.469362 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:20.470504 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:20.453857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.98.61:44439)
0619 20:59:20.453894 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.453896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.459314 (+ 5418us) server_negotiation.cc:407] Connection header received
0619 20:59:20.459455 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.459456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.459466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.459511 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.460416 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.461141 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.465699 (+ 4558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:20.465998 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:20.466036 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:20.467534 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:20.467535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:20.467536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:20.467542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:20.469286 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.469674 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.469675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.469676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.469726 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:20.469909 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:20.469910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:20.469910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:20.470022 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:20.470102 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:20.470339 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 20:59:20.470413 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:59:20.471859 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [term 1 FOLLOWER]: Refusing update from remote peer 5cfb1166e1904ae69d926d9b985b7cdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:59:20.472122 22698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [term 1 FOLLOWER]: Refusing update from remote peer 5cfb1166e1904ae69d926d9b985b7cdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:59:20.472430 22833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:59:20.472776 22833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:59:20.474045 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfb1166e1904ae69d926d9b985b7cdf. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.474133 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:20.475911 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfb1166e1904ae69d926d9b985b7cdf. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.476004 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:20.479537 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.479643 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:20.479668 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.479744 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:20.479996 22857 mvcc.cc:205] Tried to move back new op lower bound from 7298673706889740288 to 7298673706808213504. Current Snapshot: MvccSnapshot[applied={T|T < 7298673706889740288}]
I20260619 20:59:20.481045 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.481127 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:20.481315 22859 mvcc.cc:205] Tried to move back new op lower bound from 7298673706889740288 to 7298673706808213504. Current Snapshot: MvccSnapshot[applied={T|T < 7298673706889740288}]
I20260619 20:59:20.481411 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cfb1166e1904ae69d926d9b985b7cdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35605 } } peers { permanent_uuid: "69775c2c6ad8488ea26146758a57a5fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44439 } } peers { permanent_uuid: "5843d76488d243d6aed24a677b104bb1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260619 20:59:20.481489 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:20.486799 22849 catalog_manager.cc:1347] Generated new cluster ID: 9a0ba70e088d49cb8845db72d7ca8298
I20260619 20:59:20.486866 22849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:20.505932 22849 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:20.506635 22849 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:20.515370 22849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cfb1166e1904ae69d926d9b985b7cdf: Generated new TSK 0
I20260619 20:59:20.515542 22849 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:59:20.543136 22653 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.542148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.0.98.62:35605)
0619 20:59:20.542189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.542192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.542445 (+ 253us) server_negotiation.cc:407] Connection header received
0619 20:59:20.542556 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.542557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.542568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.542613 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.543070 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56948: BlockingRecv error: recv got EOF from 127.0.0.1:56948 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 20:59:20.543359 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.542144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.0.98.61:44439)
0619 20:59:20.542189 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.542192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.542763 (+ 571us) server_negotiation.cc:407] Connection header received
0619 20:59:20.542882 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.542883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.542894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.542924 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.543297 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46488: BlockingRecv error: recv got EOF from 127.0.0.1:46488 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:20.543923 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.542499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.98.60:32819)
0619 20:59:20.542538 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.542540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.543355 (+ 815us) server_negotiation.cc:407] Connection header received
0619 20:59:20.543453 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.543455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.543465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.543494 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.543864 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47752: BlockingRecv error: recv got EOF from 127.0.0.1:47752 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:20.545837 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.545227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.0.98.61:44439)
0619 20:59:20.545269 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.545271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.545316 (+ 45us) server_negotiation.cc:407] Connection header received
0619 20:59:20.545399 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.545400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.545417 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.545483 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.545792 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46502: BlockingRecv error: recv got EOF from 127.0.0.1:46502 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:20.548363 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.547850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.0.98.61:44439)
0619 20:59:20.547884 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.547885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.547893 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:59:20.547913 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.547914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.547923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.547993 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.548316 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46514: BlockingRecv error: recv got EOF from 127.0.0.1:46514 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:59:20.551750 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.551157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.98.61:44439)
0619 20:59:20.551195 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.551196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.551323 (+ 127us) server_negotiation.cc:407] Connection header received
0619 20:59:20.551354 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.551355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.551364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.551432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.551713 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46518: BlockingRecv error: recv got EOF from 127.0.0.1:46518 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:59:20.557554 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.556990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.0.98.61:44439)
0619 20:59:20.557030 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.557032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.557104 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:59:20.557166 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.557167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.557177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.557237 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.557516 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46530: BlockingRecv error: recv got EOF from 127.0.0.1:46530 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:20.567432 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.566769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.0.98.61:44439)
0619 20:59:20.566809 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.566811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.566925 (+ 114us) server_negotiation.cc:407] Connection header received
0619 20:59:20.566975 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.566976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.566987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.567046 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.567387 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46542: BlockingRecv error: recv got EOF from 127.0.0.1:46542 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:20.585228 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.584514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.0.98.61:44439)
0619 20:59:20.584551 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.584553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.584695 (+ 142us) server_negotiation.cc:407] Connection header received
0619 20:59:20.584744 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.584744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.584755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.584831 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.585174 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46554: BlockingRecv error: recv got EOF from 127.0.0.1:46554 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:59:20.619239 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.618462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.98.61:44439)
0619 20:59:20.618515 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.618517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.618652 (+ 135us) server_negotiation.cc:407] Connection header received
0619 20:59:20.618730 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.618731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.618743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.618794 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.619179 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46566: BlockingRecv error: recv got EOF from 127.0.0.1:46566 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:59:20.685029 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.684349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.98.61:44439)
0619 20:59:20.684422 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.684424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.684520 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:59:20.684559 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.684560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.684570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.684633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.684944 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46570: BlockingRecv error: recv got EOF from 127.0.0.1:46570 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 20:59:20.814952 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:20.814154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.0.98.61:44439)
0619 20:59:20.814251 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:20.814254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:20.814369 (+ 115us) server_negotiation.cc:407] Connection header received
0619 20:59:20.814470 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:20.814471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:20.814482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:20.814562 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:20.814888 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46572: BlockingRecv error: recv got EOF from 127.0.0.1:46572 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 20:59:21.072768 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:21.072107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.98.61:44439)
0619 20:59:21.072194 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.072196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.072208 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:59:21.072256 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.072257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.072274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.072343 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:21.072692 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46574: BlockingRecv error: recv got EOF from 127.0.0.1:46574 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 20:59:21.369629 22816 catalog_manager.cc:1259] Loaded cluster ID: 9a0ba70e088d49cb8845db72d7ca8298
I20260619 20:59:21.369735 22816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: loading cluster ID for follower catalog manager: success
I20260619 20:59:21.370527 22816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: acquiring CA information for follower catalog manager: success
I20260619 20:59:21.370847 22816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5843d76488d243d6aed24a677b104bb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:59:21.408640 22830 catalog_manager.cc:1259] Loaded cluster ID: 9a0ba70e088d49cb8845db72d7ca8298
I20260619 20:59:21.408717 22830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: loading cluster ID for follower catalog manager: success
I20260619 20:59:21.409543 22830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: acquiring CA information for follower catalog manager: success
I20260619 20:59:21.409852 22830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69775c2c6ad8488ea26146758a57a5fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:59:21.588579 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:21.585991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.0.98.61:44439)
0619 20:59:21.586108 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.586110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.586126 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:59:21.586196 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.586197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.586322 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.586394 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:21.586528 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.587038 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.587815 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.588002 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.588055 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:21.588184 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:21.588288 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:21.588344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:21.588451 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:59:21.588473 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260619 20:59:21.592939 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:21.590523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.0.98.60:32819)
0619 20:59:21.590648 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.590650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.590657 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:59:21.590699 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.590700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.590730 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.590811 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:21.590929 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.591393 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.592130 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.592346 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.592420 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:21.592613 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:21.592706 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:21.592793 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:21.592831 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:59:21.592854 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":48,"threads_started":1}
I20260619 20:59:21.597522 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:21.594855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.98.60:32819)
0619 20:59:21.594896 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.594898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.594905 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:59:21.594924 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.594925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.594935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.594985 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:21.595160 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.595859 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.596856 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.597064 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.597110 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:21.597261 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:21.597297 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:21.597359 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:21.597419 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 20:59:21.597439 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:59:21.598038 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:21.594522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.98.62:35605)
0619 20:59:21.594630 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.594632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.594642 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:59:21.594730 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.594731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.594780 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.594888 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:21.595201 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.595728 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.597143 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.597340 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.597416 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:21.597606 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:21.597745 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:21.597820 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:21.597930 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:59:21.597957 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260619 20:59:21.598089 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:21.594530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.61:44439)
0619 20:59:21.594568 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:21.594570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:21.594668 (+ 98us) server_negotiation.cc:407] Connection header received
0619 20:59:21.594768 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:21.594768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:21.594778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:21.594808 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:21.594973 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.595750 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.596571 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:21.596785 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:21.596865 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:21.597591 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:21.597620 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:21.597647 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:21.597997 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 20:59:21.598017 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:59:21.599658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22581
I20260619 20:59:21.604944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22656
I20260619 20:59:21.609449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22731
2026-06-19T20:59:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2903 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:59:21 dist-test-slave-8bgz krb5kdc[22945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:59:21 dist-test-slave-8bgz krb5kdc[22945](info): set up 2 sockets
Jun 19 20:59:21 dist-test-slave-8bgz krb5kdc[22945](info): commencing operation
krb5kdc: starting...
W20260619 20:59:22.646413 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/test-user.keytab.
Jun 19 20:59:22 dist-test-slave-8bgz krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:59:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:59:22.757180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:59:22.835520 22958 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:22.835723 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:22.835740 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:22.835759 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:22.836889 22958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:22.836930 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:22.836943 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:22.836956 22958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:22.837003 22958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:22.838474 22958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:22.838727 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:22.838973 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:59:22.843710 22958 server_base.cc:1034] running on GCE node
W20260619 20:59:22.843668 22964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:22.843681 22966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:22.843681 22963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:22.844172 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:22.844470 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:22.845634 22958 hybrid_clock.cc:648] HybridClock initialized: now 1781902762845620 us; error 28 us; skew 500 ppm
Jun 19 20:59:22 dist-test-slave-8bgz krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:22.848073 22958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:59:22.848507 22958 webserver.cc:460] Webserver started at http://127.0.98.62:44287/ using document root <none> and password file <none>
I20260619 20:59:22.848723 22958 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:22.848788 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:22.848930 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:22.849920 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data/instance:
uuid: "1efcd67cb90f45d0993cba8098514aa7"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "3e8b0f36c5ddf45a0edaff262b1ae861"
server_key_iv: "2d7585e35a539232817df0a89cc72587"
server_key_version: "encryptionkey@0"
I20260619 20:59:22.850334 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance:
uuid: "1efcd67cb90f45d0993cba8098514aa7"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "3e8b0f36c5ddf45a0edaff262b1ae861"
server_key_iv: "2d7585e35a539232817df0a89cc72587"
server_key_version: "encryptionkey@0"
I20260619 20:59:22.851572 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:59:22.852249 22973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:22.852416 22958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:22.852509 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal
uuid: "1efcd67cb90f45d0993cba8098514aa7"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "3e8b0f36c5ddf45a0edaff262b1ae861"
server_key_iv: "2d7585e35a539232817df0a89cc72587"
server_key_version: "encryptionkey@0"
I20260619 20:59:22.852591 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:22.873868 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:22.875626 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:22.875801 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:22.879585 22958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44345
I20260619 20:59:22.879616 23025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44345 every 8 connection(s)
I20260619 20:59:22.879951 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:59:22.880878 23026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:22.881345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 22958
I20260619 20:59:22.881445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-0/wal/instance
I20260619 20:59:22.881632 392 external_mini_cluster.cc:1428] Setting key 14a1251ceff7de7024f0d50c0130c24b
I20260619 20:59:22.882545 23026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 20:59:22 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:22.895118 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.883357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.98.62:44345)
0619 20:59:22.883490 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:22.883499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:22.883625 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:59:22.884258 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:22.884261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:22.884280 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:22.884379 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:22.884888 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:22.885697 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:22.886645 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:22.887004 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:22.887134 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:22.891690 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:22.891706 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:22.891706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:22.891714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:22.893586 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:22.893966 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:22.893969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:22.893969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:22.894055 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:22.894318 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:22.894319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:22.894319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:22.894520 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:22.894611 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:22.894798 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 20:59:22.894904 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260619 20:59:22.895121 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.883295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44345 (local address 127.0.0.1:43470)
0619 20:59:22.883485 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:22.883526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:22.884096 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:22.884384 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:22.884386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:22.884639 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:22.884782 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:22.884786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:22.885807 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:22.885808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:22.886552 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:22.886554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:22.887122 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:22.887123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:22.887186 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:22.887684 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:22.887691 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:22.891584 (+ 3893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:22.893677 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:22.893683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:22.893685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:22.893871 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:22.894127 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:22.894128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:22.894128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:22.894266 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:22.894591 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:22.894595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:22.894665 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 20:59:22.894763 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 20:59:22.894960 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":73,"thread_start_us":82,"threads_started":1}
I20260619 20:59:22.895994 23026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:22.896355 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.896250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35012)
0619 20:59:22.896298 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:22.896337 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:59:22.896503 22977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
I20260619 20:59:22.896925 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.896842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35028)
0619 20:59:22.896886 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:22.896910 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:59:22.897044 23026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:59:22.905288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40943
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:59:22.946254 23026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } attempt: 1
I20260619 20:59:22.946753 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.946612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35036)
0619 20:59:22.946677 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:22.946723 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:59:22.947194 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:22.947107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35052)
0619 20:59:22.947151 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:22.947177 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:59:22.947307 23026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40943: connect: Connection refused (error 111)
W20260619 20:59:22.984076 23033 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:22.984297 23033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:22.984326 23033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:22.984369 23033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:22.985460 23033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:22.985524 23033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:22.985551 23033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:22.985608 23033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:22.985639 23033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:22.987157 23033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:22.987417 23033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:22.987638 23033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:22.992424 23038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:22.992527 23033 server_base.cc:1034] running on GCE node
W20260619 20:59:22.992429 23041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:22.992413 23039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:22.992883 23033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:22.993196 23033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:22.994354 23033 hybrid_clock.cc:648] HybridClock initialized: now 1781902762994332 us; error 37 us; skew 500 ppm
Jun 19 20:59:22 dist-test-slave-8bgz krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:22.996757 23033 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:59:22.997215 23033 webserver.cc:460] Webserver started at http://127.0.98.61:39495/ using document root <none> and password file <none>
I20260619 20:59:22.997438 23033 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:22.997509 23033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:22.997664 23033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:22.998728 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data/instance:
uuid: "f3cec09184944faea23a10d6401a1ed5"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "949e648624316bd7da7729702fee88f2"
server_key_iv: "b8572f61bb286f1708066c00fdce9425"
server_key_version: "encryptionkey@0"
I20260619 20:59:22.999115 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal/instance:
uuid: "f3cec09184944faea23a10d6401a1ed5"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "949e648624316bd7da7729702fee88f2"
server_key_iv: "b8572f61bb286f1708066c00fdce9425"
server_key_version: "encryptionkey@0"
I20260619 20:59:23.000372 23033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:59:23.001078 23048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:23.001250 23033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:59:23.001345 23033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal
uuid: "f3cec09184944faea23a10d6401a1ed5"
format_stamp: "Formatted at 2026-06-19 20:59:22 on dist-test-slave-8bgz"
server_key: "949e648624316bd7da7729702fee88f2"
server_key_iv: "b8572f61bb286f1708066c00fdce9425"
server_key_version: "encryptionkey@0"
I20260619 20:59:23.001426 23033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:23.007700 23033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:23.009900 23033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:23.010056 23033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:23.013837 23033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40943
I20260619 20:59:23.013868 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40943 every 8 connection(s)
I20260619 20:59:23.014178 23033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:59:23.015108 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:23.015463 23026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } attempt: 2
I20260619 20:59:23.017105 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:23.019373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 23033
I20260619 20:59:23.019471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-1/wal/instance
I20260619 20:59:23.019678 392 external_mini_cluster.cc:1428] Setting key beb44eac0e1b41fdf05d035a05c4a2d8
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:23.025602 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.017730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.0.98.62:44345)
0619 20:59:23.017772 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.017774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.017923 (+ 149us) server_negotiation.cc:407] Connection header received
0619 20:59:23.017971 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.017972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.017984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.018032 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.018507 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.018993 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.019948 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.020292 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.020366 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.022623 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.022624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.022625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.022632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.024212 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.024677 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.024678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.024678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.024735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.024980 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.024981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.024981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.025097 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.025172 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.025427 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.025503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:59:23.025923 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.017671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44345 (local address 127.0.0.1:43476)
0619 20:59:23.017826 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.017855 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.017922 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.018053 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.018055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.018348 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.018416 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.018417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.019088 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.019089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.019849 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.019852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.020413 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.020414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.020493 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.020930 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.020939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.022527 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.024295 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.024302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.024304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.024603 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.024807 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.024808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.024808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.024913 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.025163 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.025169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.025300 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.025394 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.025531 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260619 20:59:23.026166 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.015740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35054)
0619 20:59:23.015808 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.015820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.015874 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.016690 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.016691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.016859 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.016901 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.016903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.018366 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.018367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.018898 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.018900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.019655 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.019656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.019733 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.020181 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.020186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.022543 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.025005 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.025006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.025006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.025236 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.025478 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.025479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.025480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.025542 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.025876 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.025878 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.025927 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.025976 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.026069 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:59:23.026569 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.015892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.0.98.61:40943)
0619 20:59:23.016047 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.016054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.016103 (+ 49us) server_negotiation.cc:407] Connection header received
0619 20:59:23.016519 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.016525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.016568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.016707 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.017002 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.018272 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.018982 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.019564 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.019684 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.022634 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.022641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.022641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.022648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.024896 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.025335 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.025338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.025338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.025396 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.025613 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.025614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.025614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.025789 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.026322 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.026372 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.026450 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260619 20:59:23.026566 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:23.027236 23026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:23.027554 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.027440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38482)
0619 20:59:23.027493 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.027524 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 20:59:23.028189 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.028108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38490)
0619 20:59:23.028145 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.028167 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 20:59:23.028299 23026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:23.034094 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.027123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.98.61:40943)
0619 20:59:23.027155 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.027156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.027165 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:59:23.027232 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.027232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.027240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.027282 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.028015 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.028625 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.029256 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.029467 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.029537 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.031622 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.031623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.031624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.031630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.032942 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.033361 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.033362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.033362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.033406 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.033596 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.033597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.033597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.033679 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.033727 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.033901 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.034011 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260619 20:59:23.034155 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.026910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35064)
0619 20:59:23.027125 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.027135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.027177 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.027293 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.027294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.027554 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.027582 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.027583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.028717 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.028718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.029177 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.029179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.029535 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.029536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.029583 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.029894 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.029898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.031548 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.033049 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.033050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.033051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.033283 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.033468 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.033469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.033470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.033537 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.033741 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.033742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.033788 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.033859 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.033966 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260619 20:59:23.034631 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:59:23.034997 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.034861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38504)
0619 20:59:23.034912 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.034968 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:59:23.035164 23050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
I20260619 20:59:23.035600 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.035476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38518)
0619 20:59:23.035534 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.035568 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 20:59:23.035719 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:59:23.046415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_bKm47/build/release/bin/kudu
/tmp/dist-test-task_bKm47/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44305
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:59:23.071452 23026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 1
I20260619 20:59:23.071921 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.071763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38526)
0619 20:59:23.071832 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.071880 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:59:23.072299 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.072209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38532)
0619 20:59:23.072252 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.072279 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:59:23.072424 23026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
I20260619 20:59:23.084950 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 1
I20260619 20:59:23.085388 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.085263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38542)
0619 20:59:23.085322 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.085360 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:59:23.085731 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.085635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38546)
0619 20:59:23.085680 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.085707 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:59:23.085822 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
I20260619 20:59:23.119602 23026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 2
I20260619 20:59:23.120090 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.119925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38548)
0619 20:59:23.120003 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.120049 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 20:59:23.120479 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.120372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38554)
0619 20:59:23.120421 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.120454 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:59:23.120585 23026 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
W20260619 20:59:23.125571 23108 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:59:23.125753 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:59:23.125770 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:59:23.125789 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:59:23.126879 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:59:23.126919 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:59:23.126932 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:59:23.126945 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:59:23.126957 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:59:23.128314 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44345,127.0.98.61:40943,127.0.98.60:44305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:59:23.128531 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:59:23.128759 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:59:23.133314 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:59:23.133335 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:23.133412 23108 server_base.cc:1034] running on GCE node
W20260619 20:59:23.133386 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:59:23.133755 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:59:23.134043 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:59:23.135226 23108 hybrid_clock.cc:648] HybridClock initialized: now 1781902763135223 us; error 38 us; skew 500 ppm
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:59:23.137535 23108 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:59:23.137887 23108 webserver.cc:460] Webserver started at http://127.0.98.60:39085/ using document root <none> and password file <none>
I20260619 20:59:23.138053 23108 fs_manager.cc:359] Metadata directory not provided
I20260619 20:59:23.138093 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:59:23.138176 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:59:23.139197 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data/instance:
uuid: "70dde80edc77475da4488a4bd2dbc21b"
format_stamp: "Formatted at 2026-06-19 20:59:23 on dist-test-slave-8bgz"
server_key: "a71bcd0cd8a60db2fa572d4795c38f6e"
server_key_iv: "cf5ce4e8a673755609cecd3ad3b7891d"
server_key_version: "encryptionkey@0"
I20260619 20:59:23.139484 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal/instance:
uuid: "70dde80edc77475da4488a4bd2dbc21b"
format_stamp: "Formatted at 2026-06-19 20:59:23 on dist-test-slave-8bgz"
server_key: "a71bcd0cd8a60db2fa572d4795c38f6e"
server_key_iv: "cf5ce4e8a673755609cecd3ad3b7891d"
server_key_version: "encryptionkey@0"
I20260619 20:59:23.140697 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:59:23.141336 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:59:23.141518 23108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:59:23.141587 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal
uuid: "70dde80edc77475da4488a4bd2dbc21b"
format_stamp: "Formatted at 2026-06-19 20:59:23 on dist-test-slave-8bgz"
server_key: "a71bcd0cd8a60db2fa572d4795c38f6e"
server_key_iv: "cf5ce4e8a673755609cecd3ad3b7891d"
server_key_version: "encryptionkey@0"
I20260619 20:59:23.141675 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:59:23.153990 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 2
I20260619 20:59:23.154424 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.154279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38564)
0619 20:59:23.154344 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.154382 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:59:23.154788 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.154688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38576)
0619 20:59:23.154736 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.154766 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:59:23.154896 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44305: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44305: connect: Connection refused (error 111)
I20260619 20:59:23.155529 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:59:23.157804 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:59:23.157959 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:59:23.161701 23108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44305
I20260619 20:59:23.161770 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44305 every 8 connection(s)
I20260619 20:59:23.162056 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:59:23.163015 23176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:59:23.164474 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:23.170950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_bKm47/build/release/bin/kudu as pid 23108
I20260619 20:59:23.171069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_bKm47/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902497118594-392-0/minicluster-data/master-2/wal/instance
I20260619 20:59:23.171293 392 external_mini_cluster.cc:1428] Setting key 8d31e726f28c2798d07d076dbfe9a544
I20260619 20:59:23.173056 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.165095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.98.62:44345)
0619 20:59:23.165151 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.165153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.165342 (+ 189us) server_negotiation.cc:407] Connection header received
0619 20:59:23.165828 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.165829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.165842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.165899 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.166511 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.166977 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.167934 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.168139 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.168221 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.170250 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.170252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.170252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.170263 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.171706 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.172093 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.172094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.172094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.172144 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.172421 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.172423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.172423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.172570 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.172646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.172885 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.172968 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:59:23.173171 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.165064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44345 (local address 127.0.0.1:43478)
0619 20:59:23.165235 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.165266 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.165721 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.165966 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.165974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.166194 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.166423 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.166427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.167078 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.167079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.167837 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.167839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.168227 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.168228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.168347 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.168845 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.168859 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.170131 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.171817 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.171821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.171822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.172028 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.172215 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.172216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.172216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.172347 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.172639 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.172643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.172756 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.172836 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.172945 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260619 20:59:23.173677 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:23.180363 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.171665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.0.98.60:44305)
0619 20:59:23.172131 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.172135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.172145 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:59:23.172161 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.172162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.172206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.172278 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.173009 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.173946 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.174757 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.175189 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.175269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.177616 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.177623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.177623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.177630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.179197 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.179546 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.179550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.179550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.179614 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.179814 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.179815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.179816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.179947 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.180093 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.180175 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.180256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":445,"thread_start_us":80,"threads_started":1}
I20260619 20:59:23.180550 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.173928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40943 (local address 127.0.0.1:35072)
0619 20:59:23.173966 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.173975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.174009 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.174129 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.174130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.174300 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.174361 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.174362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.175165 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.175166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.175898 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.175901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.176249 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.176250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.176333 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.176641 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.176646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.177897 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.179570 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.179571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.179571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.179756 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.179980 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.179981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.179981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.180035 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.180290 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.180290 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.180328 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.180390 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.180469 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260619 20:59:23.180552 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.173985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.98.61:40943)
0619 20:59:23.174026 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.174028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.174036 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:59:23.174059 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.174060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.174069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.174125 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.174456 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.175078 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.175983 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.176175 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.176232 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.178001 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.178003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.178003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.178011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.179467 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.179841 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.179842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.179843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.179897 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.180105 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.180106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.180107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.180230 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.180296 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.180416 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.180475 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:59:23.181123 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:59:23.187965 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.181335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38592)
0619 20:59:23.181392 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.181400 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.181435 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.181543 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.181543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.181678 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.181723 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.181725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.182550 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.182551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.183259 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.183261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.183617 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.183617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.183653 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.184027 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.184032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.185249 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.186952 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.186952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.186953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.187151 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.187363 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.187364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.187364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.187424 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.187659 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.187660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.187696 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.187766 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.187866 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 20:59:23.188015 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.181364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.98.60:44305)
0619 20:59:23.181396 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.181398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.181426 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:59:23.181475 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.181476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.181483 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.181535 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.181790 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.182475 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.183333 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.183548 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.183609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.185327 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.185328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.185328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.185334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.186869 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.187233 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.187234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.187234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.187279 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.187503 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.187503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.187503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.187588 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.187669 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.187794 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.187908 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":15}
I20260619 20:59:23.188843 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.181156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.98.62:44345)
0619 20:59:23.181187 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.181188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.181303 (+ 115us) server_negotiation.cc:407] Connection header received
0619 20:59:23.181363 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.181364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.181372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.181426 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.181959 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.182630 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.183499 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.183791 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.183852 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.185915 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.185916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.185916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.185923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.187585 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.187908 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.187909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.187909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.187965 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.188157 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.188159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.188159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.188295 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.188439 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.188677 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.188756 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 20:59:23.189767 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: Bootstrap starting.
I20260619 20:59:23.190706 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:23.191038 23176 log.cc:824] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:23.191834 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: No bootstrap required, opened a new log
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:59:23.194051 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.194162 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70dde80edc77475da4488a4bd2dbc21b, State: Initialized, Role: FOLLOWER
I20260619 20:59:23.194273 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.194593 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:23.194600 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.194742 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:59:23.196409 23193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:23.196466 23193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:23.197000 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.189419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.98.61:40943)
0619 20:59:23.189450 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.189452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.189460 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:59:23.189519 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.189519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.189526 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.189580 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.190099 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.190576 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.191490 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.191686 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.191749 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.194098 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.194099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.194100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.194107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.195860 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.196231 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.196232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.196233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.196293 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.196521 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.196522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.196523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.196633 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.196786 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.196829 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.196896 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:59:23.198263 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:59:23.196723 23108 master_runner.cc:186] Error getting master registration for 127.0.98.62:44345: OK, instance_id { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" instance_seqno: 1781902762878519 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:59:23.199083 23108 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:59:23.199150 23000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1efcd67cb90f45d0993cba8098514aa7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43494
I20260619 20:59:23.227752 23026 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 3
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:23.235004 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.228112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38596)
0619 20:59:23.228162 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.228196 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.228297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.228444 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.228445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.228611 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.228656 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.228657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.229328 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.229330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.229817 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.229818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.230169 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.230170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.230262 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.230751 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.230756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.231981 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.233727 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.233727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.233728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.233894 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.234111 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.234112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.234113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.234200 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.234667 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.234668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.234724 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.234805 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.234904 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 20:59:23.235019 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.228207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.0.98.60:44305)
0619 20:59:23.228311 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.228313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.228324 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:59:23.228353 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.228354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.228365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.228439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.228750 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.229241 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.229911 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.230097 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.230172 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.232062 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.232064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.232064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.232077 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.233648 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.233967 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.233968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.233969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.234022 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.234299 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.234300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.234300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.234496 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.234560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.234846 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.234924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 20:59:23.236639 23026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: Bootstrap starting.
I20260619 20:59:23.237375 23026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:23.237673 23026 log.cc:824] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:23.238476 23026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: No bootstrap required, opened a new log
I20260619 20:59:23.239856 23026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.239998 23026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efcd67cb90f45d0993cba8098514aa7, State: Initialized, Role: FOLLOWER
I20260619 20:59:23.240068 23026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.240374 23026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:23.240442 23000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:59:23.240489 23000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:23.241335 23000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.241606 23000 leader_election.cc:288] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [CANDIDATE]: Term 1 election: Requested vote from peers f3cec09184944faea23a10d6401a1ed5 (127.0.98.61:40943), 70dde80edc77475da4488a4bd2dbc21b (127.0.98.60:44305)
I20260619 20:59:23.242251 23150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efcd67cb90f45d0993cba8098514aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70dde80edc77475da4488a4bd2dbc21b"
I20260619 20:59:23.242362 23150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [term 0 FOLLOWER]: Advancing to term 1
W20260619 20:59:23.243217 23207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:23.243263 23207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:23.243290 23150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efcd67cb90f45d0993cba8098514aa7 in term 1.
I20260619 20:59:23.243501 22975 leader_election.cc:302] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efcd67cb90f45d0993cba8098514aa7, 70dde80edc77475da4488a4bd2dbc21b; no voters:
I20260619 20:59:23.243903 23213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:59:23.244091 23213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [term 1 LEADER]: Becoming Leader. State: Replica: 1efcd67cb90f45d0993cba8098514aa7, State: Running, Role: LEADER
I20260619 20:59:23.244215 23213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.244767 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efcd67cb90f45d0993cba8098514aa7. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.244953 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:23.245239 23216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:59:23.245436 23216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:59:23.243399 22958 master_runner.cc:186] Error getting master registration for 127.0.98.61:40943: OK, instance_id { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" instance_seqno: 1781902763012780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:59:23.245800 22958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:59:23.241803 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.245909 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:23.246028 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } attempt: 3
I20260619 20:59:23.247615 23150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [term 1 FOLLOWER]: Refusing update from remote peer 1efcd67cb90f45d0993cba8098514aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:59:23.247960 23214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:59:23.249202 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efcd67cb90f45d0993cba8098514aa7. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.249318 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 20:59:23 dist-test-slave-8bgz krb5kdc[22945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:59:23.253439 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.253525 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:23.254827 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.246408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44305 (local address 127.0.0.1:38598)
0619 20:59:23.246454 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:59:23.246464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:59:23.246508 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:59:23.246685 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:59:23.246686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:59:23.246854 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:59:23.246900 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.246903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.247759 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.247760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.248558 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:59:23.248561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.249397 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.249398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:59:23.249491 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.249965 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:59:23.249971 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:59:23.251518 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:59:23.253585 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.253585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.253586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.253803 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.254046 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:59:23.254047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:59:23.254047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:59:23.254121 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:59:23.254471 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:59:23.254472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:59:23.254530 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 20:59:23.254599 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:59:23.254715 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:59:23.255179 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.246332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.0.98.60:44305)
0619 20:59:23.246367 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.246369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.246560 (+ 191us) server_negotiation.cc:407] Connection header received
0619 20:59:23.246582 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.246582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.246594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.246639 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.246988 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.247657 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.248749 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.249042 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.249104 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.251601 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.251602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.251603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.251609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.253324 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.253890 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.253891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.253892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.253967 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.254198 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.254199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.254200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.254362 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.254426 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.255015 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.255097 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:59:23.256634 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: Bootstrap starting.
I20260619 20:59:23.257355 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: Neither blocks nor log segments found. Creating new log.
I20260619 20:59:23.257691 23101 log.cc:824] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:59:23.258803 23216 catalog_manager.cc:1347] Generated new cluster ID: 61b068e14594421481be41054f558fe3
I20260619 20:59:23.258857 23216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:59:23.263504 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.263641 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:23.268054 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.249255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.61:40943)
0619 20:59:23.249308 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.249310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.258785 (+ 9475us) server_negotiation.cc:407] Connection header received
0619 20:59:23.258919 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.258920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.258942 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.258971 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.259802 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.260516 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.261815 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.262098 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.262132 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.264523 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.264537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.264537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.264545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.266318 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.266980 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.266981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.266982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.267060 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.267356 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.267357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.267357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.267489 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.267561 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.267871 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.267956 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:59:23.269871 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.249691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.0.98.60:44305)
0619 20:59:23.249884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.249886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.262336 (+ 12450us) server_negotiation.cc:407] Connection header received
0619 20:59:23.262431 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.262432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.262443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.262483 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.263085 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.264151 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.265132 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.265457 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.265494 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.266975 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.266976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.266977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.266983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.268676 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.269049 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.269050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.269051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.269112 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.269330 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.269330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.269331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.269449 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.269547 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.269655 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.269730 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
I20260619 20:59:23.270176 23218 mvcc.cc:205] Tried to move back new op lower bound from 7298673718260174848 to 7298673718249619456. Current Snapshot: MvccSnapshot[applied={T|T < 7298673718260174848}]
I20260619 20:59:23.277431 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: No bootstrap required, opened a new log
I20260619 20:59:23.279449 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.279600 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3cec09184944faea23a10d6401a1ed5, State: Initialized, Role: FOLLOWER
I20260619 20:59:23.279692 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } }
I20260619 20:59:23.280048 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:59:23.281339 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1efcd67cb90f45d0993cba8098514aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f3cec09184944faea23a10d6401a1ed5"
I20260619 20:59:23.281452 23075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:59:23.282619 23075 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efcd67cb90f45d0993cba8098514aa7 in term 1.
I20260619 20:59:23.282886 23074 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [term 1 FOLLOWER]: Refusing update from remote peer 1efcd67cb90f45d0993cba8098514aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:59:23.283177 23196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:59:23.283427 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:23.243503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.0.98.62:44345)
0619 20:59:23.243540 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.243542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.250080 (+ 6538us) server_negotiation.cc:407] Connection header received
0619 20:59:23.250173 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.250174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.250184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.250231 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.251042 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.251721 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.253446 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:23.253741 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:23.253787 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:23.256586 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:59:23.256587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:59:23.256587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:59:23.256594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:59:23.258604 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.275804 (+ 17200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.275818 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.275818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.275884 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:59:23.276565 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:59:23.276566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:59:23.276566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:59:23.276699 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:59:23.276777 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:23.283242 (+ 6465us) server_negotiation.cc:299] Negotiation successful
0619 20:59:23.283337 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:59:23.286442 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.286863 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:23.286924 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1efcd67cb90f45d0993cba8098514aa7. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.286974 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:59:23.287009 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.287055 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:59:23.287539 23238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:59:23.287595 23238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:59:23.287981 23240 mvcc.cc:205] Tried to move back new op lower bound from 7298673718260174848 to 7298673718249619456. Current Snapshot: MvccSnapshot[applied={T|T < 7298673718260174848}]
I20260619 20:59:23.288813 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1efcd67cb90f45d0993cba8098514aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efcd67cb90f45d0993cba8098514aa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44345 } } peers { permanent_uuid: "f3cec09184944faea23a10d6401a1ed5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40943 } } peers { permanent_uuid: "70dde80edc77475da4488a4bd2dbc21b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44305 } } }
I20260619 20:59:23.288908 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:59:23.289235 23216 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:59:23.289975 23216 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:59:23.296983 23216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1efcd67cb90f45d0993cba8098514aa7: Generated new TSK 0
I20260619 20:59:23.297216 23216 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:59:23.328330 23223 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.326580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.0.98.60:44305)
0619 20:59:23.326632 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.326635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.327600 (+ 965us) server_negotiation.cc:407] Connection header received
0619 20:59:23.327719 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.327720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.327734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.327769 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.328260 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38618: BlockingRecv error: recv got EOF from 127.0.0.1:38618 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 20:59:23.329131 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.326839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.98.62:44345)
0619 20:59:23.326879 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.326881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.328579 (+ 1698us) server_negotiation.cc:407] Connection header received
0619 20:59:23.328683 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.328684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.328694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.328726 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.329072 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43506: BlockingRecv error: recv got EOF from 127.0.0.1:43506 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:23.329905 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.327331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.98.61:40943)
0619 20:59:23.327374 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.327377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.329366 (+ 1989us) server_negotiation.cc:407] Connection header received
0619 20:59:23.329473 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.329475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.329486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.329519 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.329840 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 20:59:23.332329 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.331517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.98.61:40943)
0619 20:59:23.331553 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.331555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.331785 (+ 230us) server_negotiation.cc:407] Connection header received
0619 20:59:23.331894 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.331895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.331912 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.331946 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.332283 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35110: BlockingRecv error: recv got EOF from 127.0.0.1:35110 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:59:23.335526 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.334549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.98.61:40943)
0619 20:59:23.334580 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.334582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.335030 (+ 448us) server_negotiation.cc:407] Connection header received
0619 20:59:23.335130 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.335131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.335141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.335171 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.335465 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35112: BlockingRecv error: recv got EOF from 127.0.0.1:35112 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:59:23.339365 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.338551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.98.61:40943)
0619 20:59:23.338584 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.338586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.338832 (+ 246us) server_negotiation.cc:407] Connection header received
0619 20:59:23.338934 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.338935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.338945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.338976 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.339314 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35116: BlockingRecv error: recv got EOF from 127.0.0.1:35116 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:59:23.345284 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.344455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.0.98.61:40943)
0619 20:59:23.344489 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.344490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.344771 (+ 281us) server_negotiation.cc:407] Connection header received
0619 20:59:23.344874 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.344874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.344884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.344915 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.345240 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35126: BlockingRecv error: recv got EOF from 127.0.0.1:35126 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:59:23.355408 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.354526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.98.61:40943)
0619 20:59:23.354565 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.354567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.354861 (+ 294us) server_negotiation.cc:407] Connection header received
0619 20:59:23.354968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.354969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.354980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.355023 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.355361 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35134: BlockingRecv error: recv got EOF from 127.0.0.1:35134 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:59:23.373783 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.372755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.0.98.61:40943)
0619 20:59:23.372802 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.372805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.373145 (+ 340us) server_negotiation.cc:407] Connection header received
0619 20:59:23.373270 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.373272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.373284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.373319 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.373722 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35138: BlockingRecv error: recv got EOF from 127.0.0.1:35138 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 20:59:23.408046 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.407204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.61:40943)
0619 20:59:23.407254 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.407258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.407425 (+ 167us) server_negotiation.cc:407] Connection header received
0619 20:59:23.407487 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.407488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.407502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.407573 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.407983 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35154: BlockingRecv error: recv got EOF from 127.0.0.1:35154 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:59:23.473935 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.473115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.0.98.61:40943)
0619 20:59:23.473200 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.473203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.473322 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:59:23.473420 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.473421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.473439 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.473504 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.473862 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35156: BlockingRecv error: recv got EOF from 127.0.0.1:35156 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 20:59:23.603863 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.603157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.0.98.61:40943)
0619 20:59:23.603259 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.603261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.603285 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:59:23.603388 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.603389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.603402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.603466 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.603803 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35170: BlockingRecv error: recv got EOF from 127.0.0.1:35170 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 20:59:23.861518 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:59:23.860896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.0.98.61:40943)
0619 20:59:23.860945 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:23.860947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:23.860972 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:59:23.861053 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:23.861054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:23.861063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:23.861110 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:59:23.861441 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35176: BlockingRecv error: recv got EOF from 127.0.0.1:35176 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:59:24.196913 23193 catalog_manager.cc:1259] Loaded cluster ID: 61b068e14594421481be41054f558fe3
I20260619 20:59:24.196973 23193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: loading cluster ID for follower catalog manager: success
I20260619 20:59:24.197724 23193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: acquiring CA information for follower catalog manager: success
I20260619 20:59:24.198047 23193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70dde80edc77475da4488a4bd2dbc21b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:59:24.288133 23238 catalog_manager.cc:1259] Loaded cluster ID: 61b068e14594421481be41054f558fe3
I20260619 20:59:24.288233 23238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: loading cluster ID for follower catalog manager: success
I20260619 20:59:24.289068 23238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: acquiring CA information for follower catalog manager: success
I20260619 20:59:24.289378 23238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3cec09184944faea23a10d6401a1ed5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:59:24.377017 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:24.374599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.0.98.61:40943)
0619 20:59:24.374735 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:24.374737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:24.374747 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:59:24.374799 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:24.374800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:24.374850 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:24.374921 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:24.375090 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.375601 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.376334 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.376546 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.376602 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:24.376731 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:24.376835 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:24.376896 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:24.376926 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 20:59:24.376944 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260619 20:59:24.381273 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:24.378983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.98.60:44305)
0619 20:59:24.379091 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:24.379092 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:24.379099 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:59:24.379135 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:24.379136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:24.379169 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:24.379239 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:24.379315 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.379847 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.380498 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.380762 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.380824 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:24.380953 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:24.381050 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:24.381104 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:24.381121 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 20:59:24.381137 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260619 20:59:24.387066 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:24.383487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.98.61:40943)
0619 20:59:24.383535 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:24.383536 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:24.383740 (+ 204us) server_negotiation.cc:407] Connection header received
0619 20:59:24.383851 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:24.383852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:24.383864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:24.383898 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:24.384122 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.384832 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.385982 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.386303 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.386347 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:24.386579 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:24.386623 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:24.386652 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:24.386978 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 20:59:24.387002 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:59:24.387588 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:24.383404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.0.98.60:44305)
0619 20:59:24.383444 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:24.383446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:24.383472 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:59:24.383642 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:24.383643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:24.383655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:24.383942 (+ 287us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:24.384021 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.385016 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.386517 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.386918 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.387050 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:24.387215 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:24.387246 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:24.387461 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:24.387482 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 20:59:24.387512 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:59:24.387888 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:59:24.383174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.0.98.62:44345)
0619 20:59:24.383290 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 20:59:24.383292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:59:24.383713 (+ 421us) server_negotiation.cc:407] Connection header received
0619 20:59:24.383839 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:59:24.383840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:59:24.383889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:59:24.383944 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:59:24.384155 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.384970 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.386237 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:59:24.386603 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:59:24.386642 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:59:24.387315 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:59:24.387427 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:59:24.387491 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:59:24.387600 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 20:59:24.387792 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260619 20:59:24.392948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 22958
I20260619 20:59:24.398391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 23033
I20260619 20:59:24.403424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_bKm47/build/release/bin/kudu with pid 23108
2026-06-19T20:59:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2793 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5697 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267291 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST